Numerous studies have investigated whether additional abilities of artificial agents improve the interaction with their users. Typically, a specific capability was added to a baseline system and participants interacted with both versions. Questionnaires (e.g., GOODSPEED, Bartneck et al., 2009) or time-related metrics combined with a questionnaire (e.g., PARADISE, Walker et al., 2000) were then evaluated. In most published work, the new capability demonstrably improved the user ratings. For example, agents with the capability to praise (Fasola and Mataric, 2012), err (Salem et al., 2013), and blink pleasantly (Takashima et al., 2008) were more likeable than their counterparts without this capability. However, subjective ratings can be unreliable (Cahill, 2009; Belz and Kow, 2010) and hardly address questions beyond likeability, e.g., whether an action is perceived as social or task-oriented, and when and why an agent is more or less comprehensible. But investigating human social behavior with controlled and ecologically valid experiments is notoriously difficult given the variance in natural interactions (e.g., Bergmann et al., 2010; Sciutti et al., 2015). Artificial agents can precisely reproduce interactive behavior in real-time but are still rarely used for investigations beyond likeability.
Conrad et al. (2015) used subjective ratings and objective metrics (gaze behavior, response accuracy, number of clarification questions and backchannels) for investigating how the users' comprehension of straightforward and complicated survey questions1 was affected by the virtual agent's facial expressiveness2 and dialogue capability3. As expected, the subjective ratings improved with more advanced agent capabilities. Objective metrics such as response accuracy and eye tracking data provided further insights, e.g., whether interviewees avoid eye contact with their interviewer (gaze aversion). Research in human-human interaction indicated that gaze aversion was not social behavior but that it reduced distraction and directed all the interviewee's cognitive resources toward answering the question (Glenberg et al., 1998; Doherty-Sneddon et al., 2002; Doherty-Sneddon and Phelps, 2005). But the interviewees' gaze patterns in Conrad et al's (2015) study were independent of the question's difficulty if the agent had high dialogue capabilities. This implies a social component in gaze aversion that cannot be ruled out by the original studies that used (a) human questioners such that gaze aversion could be social (Glenberg et al., 1998; Doherty-Sneddon et al., 2002), (b) no questioner such that truly social gaze behavior was impossible to observe (Glenberg et al., 1998), or (c) memory tasks (Glenberg et al., 1998; Doherty-Sneddon and Phelps, 2005) that are sensitive to any intervening information such as faces (Posner and Konick, 1966; Dale, 1973; West, 1999; de Fockert et al., 2001). A social component in gaze aversion is not surprising given the social nature of other behaviors. For example, facial expressions of surprise were more pronounced if other humans were present in the room (Schützwohl and Reisenzein, 2012). Even the electrodermal activity of truth tellers and deceivers differed only if they believed their virtual interviewer was controlled by human but not by artificial intelligence (Ströfer et al., 2016). Thus, gaze aversion appears to be at least partly social. Conrad et al's (2015) findings illustrate that objective metrics in HMI reveal new and additional insights (cf. Zarrieß et al., 2015) providing a valuable tool for understanding social interaction.
Designing artificial agents driven by likeability tends to accumulate capabilities regardless of their contribution to the interaction. First, (likeable) capabilities contribute differently to an interaction and do not necessarily increase an artificial agent's comprehensibility. Conrad et al. (2015) tested the effect of each capability on their participants' behavior. For example, high dialogue capabilities produced the same response accuracy irrespectively of the agent's facial expressiveness. On the other hand, facial expressiveness influenced the participants' back-channeling behavior which could smoothen the interaction (cf. Clark and Brennan, 1991; Clark and Krych, 2004). Secondly, simply combining likeable behaviors does not necessarily result in a human-like, comprehensible artificial agent. For example, humans have difficulty integrating human beat gestures and speech if enacted by a robot (Bremner and Leonards, 2015). Also, averaging across behaviors of individuals may result in an unintelligible chimera whereas humans prefer the behavior of one individual (Bergmann et al., 2010; Kelly et al., 2010). Arguably, a fictional character (or artificial agent) gains appeal by focussing on essential communicative behaviors that form a coherent, expressive and comprehensible individual (Thomas and Johnston, 1995). Thirdly, not being perceived as human may not be likeable but advantageous because users would assume an automatic agent operates consistently rather than having bad intentions or preferring someone over someone else. Also, this intuitively differentiates between humans and artificial agents (cf. Stein and Ohler, 2017). In contrast to being all human-like, using human cognitive principles for maximizing an agent's capability to interact with its users is highly desirable. Thus, objective metrics and insightfully designed experiments are more important than likeable capabilities.
In sum, Conrad et al. (2015) estimated the communicative relevance of an artificial agent's social behaviors given a specific task with objective metrics such as response accuracy and gaze patterns. Their study demonstrated that the difficult research in social interaction can benefit from artificial agents if combined with insightful, reliable, replicable and objective dependent metrics.
Statements
Author contributions
The author confirms being the sole contributor of this work and approved it for publication.
Acknowledgments
I would like to thank my colleagues Christina Unger, Hendrik Buschmeier, and Julian Hough for providing their comments on this manuscript.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Footnotes
1.^Questions from the US current population survey (monthly survey of 60,000 households estimating, e.g., unemployment rate, see United States Census Bureau, 2016) were “complicated” if they were ambiguous (e.g., is a floor lamp a piece of furniture or an electrical appliance) and likely triggered clarification questions.
2.^Facial animation was either low (head and face did not move, the eyes did not blink and the mouth was only opened and closed indicating whether the agent was speaking) or high (21 channel-motion tracking of a human face, appropriate lip movements, blinking behavior at all times during the interview).
3.^Dialogue capability was either low (the agent responded to user questions with a neutral probe or repeated the interview question) or high (the agent tried to provide helpful answers such as clarifying terms). In both conditions, the agent read out the interview questions and repeated them if needed.
References
1
BartneckC.KulicD.CroftE.ZoghbiS. (2009). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int. J. Soc. Robot.1, 71–81. 10.1007/s12369-008-0001-3
2
BelzA.KowE. (2010). Comparing rating scales and preference judgements in language evaluation, in Proceedings of the 6th International Natural Language Generation Conference (Dublin: Association for Computational Linguistics), 7–15.
3
BergmannK.KoppS.EysselF. (2010). Individualized gesturing outperforms average gesturing - evaluating gesture production in virtual humans, in Proceedings of the 10th International Conference on Intelligent Virtual Agents (LNCS 6356), eds AllbeckJ.BadlerN.BickmoreT.PelachaudC.SafonovaA. (Berlin; Heidelberg: Springer International Publishing), 104–117.
4
BremnerP.LeonardsU. (2015). Speech and gesture emphasis effects for robotic and human communicators: a direct comparison, in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (Portland, OR: ACM Press), 255–262. 10.1145/2696454.2696496
5
CahillA. (2009). Correlating human and automatic evaluation of a German surface realiser, in ACLShort '09 Proceedings of the ACL-IJCNLP 2009 (Singapore: Association for Computational Linguistics), 97–100.
6
ClarkH. H.BrennanS. E. (1991). Grounding in communication, in Perspectives on Socially Shared Cognition, 1st Edn., eds ResnickL. B.LevineJ. M.TeasleyS. D. (Washington, DC: American Psychological Association), 127–149.
7
ClarkH. H.KrychM. A. (2004). Speaking while monitoring addressees for understanding. J. Mem. Lang.50, 62–81. 10.1016/j.jml.2003.08.004
8
ConradF. G.SchoberM. F.JansM.OrlowskiR. A.NielsenD.LevensteinR. (2015). Comprehension and engagement in survey interviews with virtual agents. Front. Psychol.6:1578. 10.3389/fpsyg.2015.01578
9
DaleH. C. A. (1973). Short-term memory for visual information. Brit. J. Psychol.64, 1–8. 10.1111/j.2044-8295.1973.tb01320.x
10
de FockertJ. W.ReesG.FrithC. D.LavieN. (2001). The role of working memory in visual selective attention. Science291, 1803–1806. 10.1126/science.1056496
11
Doherty-SneddonG.BruceV.BonnerL.LongbothamS.DoyleC. (2002). Development of gaze aversion as disengagement from visual information. Dev. Psychol.38, 438–445. 10.1037/0012-1649.38.3.438
12
Doherty-SneddonG.PhelpsF. G. (2005). Gaze aversion: a response to cognitive or social difficulty?Mem. Cogn.33, 727–733. 10.3758/BF03195338
13
FasolaJ.MataricM. J. (2012). Using socially assistive human-robot interaction to motivate physical exercise for older adults. Proc. IEEE100, 2512–2526. 10.1109/JPROC.2012.2200539
14
GlenbergA. M.SchroederJ. L.RobertsonD. A. (1998). Averting the gaze disengages the environment and facilitates remembering. Mem. Cogn.26, 651–658. 10.3758/BF03211385
15
KellyS. D.CreighP.BartolottiJ. (2010). Integrating speech and iconic gestures in a stroop-like task: evidence for automatic processing. J. Cogn. Neurosci.22, 683–694. 10.1162/jocn.2009.21254
16
PosnerM. I.KonickA. F. (1966). Short-term retention of visual and kinesthetic information. Organ. Behav. Hum. Perform.1, 71–86. 10.1016/0030-5073(66)90006-7
17
SalemM.EysselF.RohlfingK.KoppS.JoublinF. (2013). To ERR is human(-like): effects of robot gesture on perceived anthropomorphism and likability. Int. J. Soc. Robot.5, 313–323. 10.1007/s12369-013-0196-9
18
SchützwohlA.ReisenzeinR. (2012). Facial expressions in response to a highly surprising event exceeding the field of vision: a test of Darwin's theory of surprise. Evol. Hum. Behav.33, 657–664. 10.1016/j.evolhumbehav.2012.04.003
19
SciuttiA.AnsuiniC.BecchioC.SandiniG. (2015). Investigating the ability to read others' intentions using humanoid robots. Front. Psychol.6:1362. 10.3389/fpsyg.2015.01362
20
SteinJ.-P.OhlerP. (2017). Venturing into the uncanny valley of mind-The influence of mind attribution on the acceptance of human-like characters in a virtual reality setting. Cognition160, 43–50. 10.1016/j.cognition.2016.12.010
21
StröferS.UfkesE. G.BruijnesM.GiebelsE.NoordzijM. L. (2016). Interviewing suspects with avatars: avatars are more effective when perceived as human. Front. Psychol.7:545. 10.3389/fpsyg.2016.00545
22
TakashimaK.OmoriY.YoshimotoY.ItohY.KitamuraY.KishinoF. (2008). Effects of avatar's blinking animation on person impressions, in GI '08 Proceedings of Graphics Interface 2008, vol. 169–176 (Windsor, ON: Canadian Information Processing Society).
23
ThomasF.JohnstonO. (1995). The Illusion of Life: Disney Animation, 1st Hyperion Edn. New York, NY: Hyperion.
24
United States Census Bureau (2016). Current Population Survey (CPS).
25
WalkerM.KammC.LitmanD. (2000). Towards developing general models of usability with PARADISE. Nat. Lang. Eng.6, 363–377. 10.1017/S1351324900002503
26
WestR. (1999). Visual distraction, working memory, and aging. Mem. Cogn.27, 1064–1072. 10.3758/BF03201235
27
ZarrießS.LothS.SchlangenD. (2015). Reading times predict the quality of generated text above and beyond human ratings, in Proceedings of the 15th European Workshop on Natural Language Generation, eds BelzA.GattA.PortetF.PurverM. (Brighton: The Association for Computational Linguistics), 38–47.
Summary
Keywords
virtual agents, social signals, interaction design, gaze aversion, social eye gaze, dialogue capability, facial animation
Citation
Loth S (2017) Beyond Likeability: Investigating Social Interactions with Artificial Agents and Objective Metrics. Front. Psychol. 8:1662. doi: 10.3389/fpsyg.2017.01662
Received
21 July 2016
Accepted
11 September 2017
Published
22 September 2017
Volume
8 - 2017
Edited and reviewed by
Claire Marie Fletcher-Flinn, University of Auckland, New Zealand
Updates
Copyright
© 2017 Loth.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Sebastian Loth sebastian.loth@uni-bielefeld.de
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.