Joint attention, defined as the coordination of orienting between two or more people toward an object, person or event (Billeci et al., 2017; Mundy and Newell, 2007; Scaife and Bruner, 1975), is one of the essential mechanisms of social interaction (Chevalier et al., 2020). This joint attention can be signalled through both verbal and visual cues, with gaze direction providing an important visual signal of attention and thus joint attention (Kleinke, 1986; Emery, 2000; Land and Tatler, 2009). Therefore, it is crucial to understand how eye gaze is used as part of joint attention in social scenarios. Importantly, the emergence of virtual agents and social robots gives us the opportunity to better understand these processes as they may occur in human-to-human interaction, as well as enabling realistic human to agent/robot interaction.

Traditionally, the influence of gaze on attentional orienting has been studied in cueing paradigms based on Posner’s original spatial task (e.g., Posner, 1980). Using simple, static face stimuli, e.g., photographs of real faces or drawings of schematic faces (see Box 1, panel A), results show that gaze provides a very strong attentional cue (Frischen et al., 2007). Importantly, though the basic cueing effect has also been replicated with non-social cues such as arrows or direction words (Hommel et al., 2001; Ristic et al., 2002; Tipples, 2002, 2008), it is argued that the strength and immediacy of gaze cuing demonstrates something special about how we respond to social information, and in particular eye gaze (Frischen et al., 2007; Kampis and Southgate, 2020; Stephenson et al., 2021). BOX 1

Examples of the range of stimuli used in research presented in this review rated on realism, flexibility (of movement) and experimental control. The left side begins with highly controlled but unrealistic and inflexible stimuli of static faces (A), next we show the poorly controlled, but flexible and potentially realistic approach of using real humans (B), though arguably realism is weakened by the contrived nature of studies and environments, with this being exacerbated by the addition of experimentally necessary but distinctive brain imaging technology in the examples shown here. Finally, on the right we arguably achieve greater realism and control, as well as reasonable movement flexibility by using dynamic virtual agents (C) where the eyes and heads can move. Presented examples have either been reproduced with permission or created for the box to be representative of stimuli used. 1. Schematic faces, e.g., Friesen and Kingstone, (1998); 2. “Dynamic” photograph gaze shift stimuli where a direct face is present prior to an averted gaze face*, e.g., Chen et al. (2021); 3. Averted gaze photograph stimuli*, e.g., Driver et al. (1999); 4. Human-human interaction, reproduced with permission from Dravida et al. (2020); 5. Human-human shared attention paradigm, reproduced with permission from Lachat et al. (2012); 6. Human as gaze cue, example used with permission from Cole et al. (2015); 7. Dynamic virtual agent head, permitted reproduction from Caruana et al. (2020); 8. Virtual agent as gaze cue (lab stimuli), as used in Gregory (2021); 9. Virtual agent as interaction partner with realistic gaze movement (lab stimuli) Kelly at al., (in prep; see video youtu.be/sgrxOpYP91E).

*Face images from The Radboud Faces database, Langner et al. (2010).

While research conducted into gaze cuing phenomena to date has been informative, the use of simplistic and most often static face stimuli such as photographs of real faces, drawings of schematic faces or even just eyes (see Box 1, panel A), as well as the frequent use of unrealistic tasks and environments (e.g., a face image “floating” in a 2D spatial environment) has been highlighted as problematic if we truly wish to understand social processes (Gobel et al., 2015; Risko et al., 2012, 2016; Zaki and Ochsner, 2009). For example, while research in traditional 2D settings shows that participants tend to focus on the eye region of faces (Birmingham et al., 2008), research conducted in real life shows participants avoiding direct eye contact (Foulsham et al., 2011; Laidlawet al., 2011; Mansour and Kuhn, 2019). Since the engagement of eye contact and the following of eye gaze direction are crucial aspects of joint attention, it is extremely important to look for alternative approaches that may better reflect real life social interaction.

One obvious option to better understand how joint attention might manifest in real world social interaction is to observe interaction between real humans. However, while observational studies can be informative, it is also important to employ experimental studies in order to build theories and test hypotheses about the nature of human interaction. Indeed, a number of empirical studies have been conducted using real human interaction, which in principle replicate known gaze cuing and joint attention effects (Cole et al., 2015; Dravida et al., 2020; Lachat et al., 2012, see Box 1, panel B). However, using real humans as interaction partners in such empirical studies has a number of important limitations which affect the kinds of studies and level of nuance possible.

First, such studies are resource heavy, requiring a confederate’s concentration and time, thus limiting the number of repetitions possible (trials and participants). Second, study design is limited; for example, it is impossible to subtly change timings of gaze shifts in the millisecond range, change the identity of the gaze cue during the task, and change other aspects of stimulus presentation. Third, experimental control is limited; real people will not perform the same action in the same way multiple times during an experimental session and are also unlikely to behave in exactly the same way in each different experimental session. Further, they may make many involuntary nuanced facial expressions such as smirking or raising their eyebrows, as well as uncontrollable micro expressions (e.g., Porter and ten Brinke, 2008; Porter et al., 2012), potentially affecting the validity of the study (e.g., Kuhlen and Brennan, 2013). Finally, while using human confederates, participants are often taking part in a very unnatural “social” experience, i.e., engaging in a highly artificial task in a lab environment while being stared at by a stranger who is not communicating in a particularly natural way (See Box 1, panel B).

Therefore, while using real humans provides real-life dynamic interaction, this comes at the cost of experimental control and design complexity. While others have suggested the use of social robots (e.g., Chevalier et al., 2020), here, we propose that virtual agents are the optimum alternative for the experimental study of joint attention.

Virtual Agents (VAs: also referred to as virtual humans or characters) are defined as computer-generated virtual reality characters with human-like appearances, in contrast to an avatar, which is a humanoid representation of a user in a virtual world (Pan and Hamilton, 2018). These VAs offer the opportunity to conduct social interaction studies with higher realism while retaining experimental control. Importantly, similar social behaviours have been found during interactions with VAs as are observed during real human interaction (for reviews and further discussion see Bombari et al., 2015; Kothgassner and Felnhofer, 2020; Pan and Hamilton, 2018).

The use of VAs is highly cost effective, with ready-made or easy to modify assets available on several platforms (Examples: Mixamo: www.mixamo.com; SketchFab: sketchfab.com; MakeHuman: www.makehumancommunity.org). These VA assets can then be manipulated for the purposes of a study using free software such as Blender for animation and Unity3D for experimental development and control. The VAs can be presented in fully immersive virtual systems through immersive VR headsets as well as through augmented virtual reality and finally through basic computer setups, each of which having pros and cons related to expense, portability and immersion (Pan and Hamilton, 2018). Importantly, even using VA-based stimuli (e.g., dynamic video recordings) in traditional screen-based studies offers significantly more social nuance and realism compared to using static images of disembodied heads (e.g., Gregory, 2021, see Box 1, panel C). Indeed, gaze is associated with an intent to act (Land and Tatler, 2009) and it is difficult to imagine a static head performing a goal directed action.

When considering using VAs as social partners, researchers may consider factors such as the realism of the VAs, including the issue of the “uncanny valley” and whether the participant perceives the VA as a social partner at all. It may be argued that people will not interact with VAs in the same way they will with real humans. However, research indicates that this concern is unfounded (Bombari et al., 2015; Kothgassner and Felnhofer, 2020; Pan and Hamilton, 2018), indeed, research in moral psychology demonstrates strong realistic response in VR (Francis et al., 2016; Francis et al., 2017; Niforatos et al., 2020).

The “uncanny valley” refers to our loss of affinity to computer-generated agents when they fail to attain human-like realism (Mori et al., 2012). This can serve as a reminder that the VA is not real, limiting the naturalness of the interaction. First, it is important to note that the same arguments of unnaturalness can also be applied to some studies using real humans, as discussed above and presented in Box 1, panel B. Therefore, even rudimentary VAs can be beneficial when investigating social interaction. Indeed, research suggests that even presenting the most basic VA can be successful if the eyes are communicative, with findings showing responses to gaze in human-to-agent interactions are comparable to those in human-human interaction (Ruhland et al., 2015).

Joint attention as initiated through eye gaze is therefore an area ripe for the use of VAs because it is generally investigated in isolation from verbal cues, as well as in isolation from body movements which can affect responses independently (e.g., Mazzarella et al., 2012), mitigating concerns regarding the complexity of producing realistic speech and action (Pan and Hamilton, 2018). Importantly, for use in joint attention research VAs can mimic the dynamic behaviour of human gaze with significant precision, with VAs having full eye movement available (e.g., Ruhland et al., 2015). Similar to whole-body motion capture for animations, it is possible to apply real-time and/or recorded human eye movements onto VAs, e.g., by using HMDs with eye-tracking such as the HTC Vive Pro Eye (See video: https://youtu.be/sgrxOpYP91E), creating the impression of a highly naturalistic interaction, but one with the key aspects of experimental control. This therefore goes significantly beyond traditional gaze cuing studies where gaze is presented statically in the desired directions (See Box 1, panel A). Importantly, VAs allow investigation of social-interaction with real human gaze sequences as well as allowing the possibility of a closed-loop system that would determine the VAs’ behaviour based on the users’ gaze (Kelly et al., 2020). This therefore allows fundamental investigations into how a person responds when their gaze is followed.

Going beyond eye gaze, VAs also enable the consistent use of age-matched stimuli, especially when investigating child development or aging, where it may be difficult to use age-matched confederates. Age matching as well as controlled age mismatching can be vital in understanding changes in joint attention and other aspects of social cognition across the lifespan (e.g., Slessor et al., 2010), as well as when investigating differences in children with learning difficulties related to ADHD and autism diagnosis (e.g., Bradley and Newbutt, 2018; Jyoti et al., 2019). VAs also facilitate investigations of cueing with a controlled variety of different “people” as well as more general effects of multiagent joint attention (Capozzi et al., 2015; Capozzi et al., 2018). Indeed, the presentation of VAs allows full control of the distance between each partner and their gaze behaviour while allowing for a natural configuration of people in a room; a difficult endeavour with multiple human confederates.

Virtual scenarios also enable the manipulation of the environment in which the interaction occurs, and of the stimuli presented as part of the task. This allows investigation of how interactions may occur differently in indoor vs outdoor environments, as well as comparing formal learning environments like a classroom to less formal environments. Consequently, future investigations of joint attention in more dynamic, interactive VR scenarios could focus on how the participant explores the virtual world(s) with their virtual social partner(s).

The use of virtual reality and agents (VAs) in studying human interaction allows researchers a high level of experimental control while allowing participants to engage naturally in a realistic experimental environment. This enables researchers to study the nuances of joint attention—and social interaction more generally—without the common pitfalls of both simplistic static face-stimuli as well as studies using real humans. In contrast to using human-to-human interaction, the functionality and control of VAs mitigates any lack of realism that may be experienced. Arguably it is more natural to interact with a VA presented in a realistic VR environment than with a real human in unusual experimental settings/headgear (see Box 1). Further, though it has been proposed that social robots are an ideal alternative to human to human interaction, particularly in terms of control and realism of eye gaze interaction (Chevalier et al., 2020), VAs, particularly when presented in fully immersive VR environments, are more versatile, more cost effective and potentially more realistic than social robots. VAs offer infinite options for quick adjustments in terms of appearance (e.g., age), environment and kinematics as well as a significantly higher control over the experimental setting. This is further corroborated by the accelerated emergence of the so-called “Metaverse” and we therefore argue that future breakthroughs in understanding human social interaction will likely come from investigations using VAs believed to have genuine agency in a task. Therefore, having VAs that can react based on a participant’s behaviour, and in particular to their gaze behaviour, opens the door to countless experiments that can provide stronger insights into how joint attention affects our cognition.