AUTHOR=Nunnemann Eva M. , Kreysa Helene , Knoeferle Pia 

TITLE=The effects of referential gaze in spoken language comprehension: Human speaker vs. virtual agent listener gaze

JOURNAL=Frontiers in Communication

VOLUME=Volume 8 - 2023

YEAR=2023

URL=https://www.frontiersin.org/journals/communication/articles/10.3389/fcomm.2023.1029157

DOI=10.3389/fcomm.2023.1029157

ISSN=2297-900X

ABSTRACT=A set of four studies addressed effects of human speaker gaze vs. virtual agent listener gaze on reaction times, accuracy and eye movements. The present article focusses on the results from the eye tracking data exclusively. Participants saw videos in which a static scene depicting three
characters was presented on a screen. Eye movements were recorded as participants listened to
German subject-verb-object (SVO) sentences describing an interaction between two of these three characters. After each trial a template schematically depicting three characters and the interaction appeared on screen. Participants verified a match between sentence and template. Two critical factors were manipulated across all four experiments: (1) whether the human speaker uttering the sentence - was visible, and (2) whether the agent listener was present. Moreover, in Experiments 2 and 4 the target noun phrase 2 (NP2) was made inaudible and in Experiments 3 and 4 the gaze time course of the agent listener was altered: it looked at the target character about 400 ms before the speaker. These manipulations served to increase the value of the speaker’s and listener’s gaze cues for correctly anticipating the target character. Human speaker gaze led to increased fixations of the target character in all experiments, but primarily after the onset of its mention. Only in Experiment 3 did participants reliably anticipate the target character prior to its mention, in this case making use of both the human speaker’s and the virtual agent listener’s gaze. In all other cases, virtual agent listener gaze had no effect on participants’ visual anticipation of the target – even in conditions in which it was the exclusive cue towards the target referent.