<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Comput. Sci.</journal-id>
<journal-title>Frontiers in Computer Science</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Comput. Sci.</abbrev-journal-title>
<issn pub-type="epub">2624-9898</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fcomp.2023.1151150</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Computer Science</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Exploring the effects of human-centered AI explanations on trust and reliance</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Scharowski</surname> <given-names>Nicolas</given-names></name>
<xref ref-type="corresp" rid="c001"><sup>&#x0002A;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/2182778/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Perrig</surname> <given-names>Sebastian A. C.</given-names></name>
<uri xlink:href="http://loop.frontiersin.org/people/2111486/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Svab</surname> <given-names>Melanie</given-names></name>
</contrib>
<contrib contrib-type="author">
<name><surname>Opwis</surname> <given-names>Klaus</given-names></name>
</contrib>
<contrib contrib-type="author">
<name><surname>Br&#x000FC;hlmann</surname> <given-names>Florian</given-names></name>
<uri xlink:href="http://loop.frontiersin.org/people/528139/overview"/>
</contrib>
</contrib-group>
<aff><institution>Human-Computer Interaction Research Group, Center for General Psychology and Methodology, Faculty of Psychology, University of Basel</institution>, <addr-line>Basel</addr-line>, <country>Switzerland</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Chathurika S. Wickramasinghe Brahmana, Capital One, United States</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Martin Gjoreski, University of Italian Switzerland, Switzerland; Antonis Bikakis, University College London, United Kingdom; Nadine Schlicker, University of Marburg, Germany</p></fn>
<corresp id="c001">&#x0002A;Correspondence: Nicolas Scharowski <email>nicolas.scharowski&#x00040;unibas.ch</email></corresp>
</author-notes>
<pub-date pub-type="epub">
<day>17</day>
<month>07</month>
<year>2023</year>
</pub-date>
<pub-date pub-type="collection">
<year>2023</year>
</pub-date>
<volume>5</volume>
<elocation-id>1151150</elocation-id>
<history>
<date date-type="received">
<day>25</day>
<month>01</month>
<year>2023</year>
</date>
<date date-type="accepted">
<day>27</day>
<month>06</month>
<year>2023</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2023 Scharowski, Perrig, Svab, Opwis and Br&#x000FC;hlmann.</copyright-statement>
<copyright-year>2023</copyright-year>
<copyright-holder>Scharowski, Perrig, Svab, Opwis and Br&#x000FC;hlmann</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p></license> </permissions>
<abstract>
<p>Transparency is widely regarded as crucial for the responsible real-world deployment of artificial intelligence (AI) and is considered an essential prerequisite to establishing trust in AI. There are several approaches to enabling transparency, with one promising attempt being human-centered explanations. However, there is little research into the effectiveness of human-centered explanations on end-users&#x00027; trust. What complicates the comparison of existing empirical work is that trust is measured in different ways. Some researchers measure subjective trust using questionnaires, while others measure objective trust-related behavior such as reliance. To bridge these gaps, we investigated the effects of two promising human-centered <italic>post-hoc</italic> explanations, <italic>feature importance</italic> and <italic>counterfactuals</italic>, on trust <italic>and</italic> reliance. We compared these two explanations with a control condition in a decision-making experiment (<italic>N</italic> = 380). Results showed that human-centered explanations can significantly increase reliance but the type of decision-making (increasing a price vs. decreasing a price) had an even greater influence. This challenges the presumed importance of transparency over other factors in human decision-making involving AI, such as potential heuristics and biases. We conclude that trust does not necessarily equate to reliance and emphasize the importance of appropriate, validated, and agreed-upon metrics to design and evaluate human-centered AI.</p></abstract>
<kwd-group>
<kwd>AI</kwd>
<kwd>XAI</kwd>
<kwd>HCXAI</kwd>
<kwd>trust</kwd>
<kwd>reliance</kwd>
<kwd>transparency</kwd>
<kwd>explainability</kwd>
<kwd>interpretability</kwd>
</kwd-group>
<counts>
<fig-count count="4"/>
<table-count count="1"/>
<equation-count count="2"/>
<ref-count count="88"/>
<page-count count="15"/>
<word-count count="12605"/>
</counts>
<custom-meta-wrap>
<custom-meta>
<meta-name>section-at-acceptance</meta-name>
<meta-value>Theoretical Computer Science</meta-value>
</custom-meta>
</custom-meta-wrap>
</article-meta>
</front>
<body>
<sec sec-type="intro" id="s1">
<title>1. Introduction</title>
<p>It is generally recognized that computers perform specific tasks better than humans, such as numeracy, logical reasoning, or storing information (Solso et al., <xref ref-type="bibr" rid="B73">2005</xref>). But with the recent breakthroughs in artificial intelligence (AI), domains that used to be exclusively associated with human competence and considered computationally unattainable are likewise being challenged by machines. AI has led to improvements in speech recognition, image classification, as well as object detection (LeCun et al., <xref ref-type="bibr" rid="B46">2015</xref>) and is now increasingly used in various everyday applications such as video surveillance, email spam filtering, online customer support, and product recommendations. Because of this general applicability and the potential manifold consequences, voices are being raised that AI should satisfy criteria like fairness, reliability, accountability, and transparency (Ehsan et al., <xref ref-type="bibr" rid="B22">2021b</xref>; ACM FAccT Conference, <xref ref-type="bibr" rid="B2">2022</xref>). The call for transparent AI has led to the multidisciplinary research field of explainable artificial intelligence (XAI), which explores methods and models that make the behaviors, predictions, and decisions of AI transparent and understandable to humans (Lipton, <xref ref-type="bibr" rid="B49">2018a</xref>; Liao et al., <xref ref-type="bibr" rid="B48">2020</xref>). Abdul et al. (<xref ref-type="bibr" rid="B1">2018</xref>), as well as Biran and Cotton (<xref ref-type="bibr" rid="B8">2017</xref>) have pointed out that the development of transparent systems has long been a research focus, originating from expert systems, intelligent agents, recommender systems, context-aware systems, and other adjacent fields such as automation.</p>
<p>Despite this rich history, current XAI research faces unprecedented challenges as AI is increasingly complex and thus more cumbersome to render transparent (Biran and Cotton, <xref ref-type="bibr" rid="B8">2017</xref>). In the pursuit of ever more accurate predictions, modern AI consists of millions of interdependent values and parameters, resulting in a trade-off between complexity and transparency (Shmueli, <xref ref-type="bibr" rid="B72">2010</xref>; Mittelstadt et al., <xref ref-type="bibr" rid="B57">2019</xref>). Because of this complexity, AI is often characterized by the opaque box paradigm (Suresh et al., <xref ref-type="bibr" rid="B76">2021</xref>), meaning that AI can only be considered in terms of its inputs and outputs without direct observations of its inner workings. This opacity makes it more challenging than ever to ensure fairness, reliability, and accountability, rendering transparency a prerequisite for the other three criteria. Some researchers argue that transparency helps verify and improve the functionality of a system (i.e., for debugging), supports developers in learning from a system (i.e., in generating hypotheses), or is needed to ensure fair and ethical decision-making (Mittelstadt et al., <xref ref-type="bibr" rid="B57">2019</xref>). Others believe that transparency contributes toward building a relationship of trust between humans and AI (Stephanidis et al., <xref ref-type="bibr" rid="B75">2019</xref>), which plays a key role in people&#x00027;s willingness to rely on automated systems (Hoff and Bashir, <xref ref-type="bibr" rid="B31">2015</xref>).</p>
<p>While transparency is generally considered crucial for the effective and responsible real-world deployment of AI, there are various transparency approaches tailored to the algorithm&#x00027;s goal, the context, and the involved stakeholders, such as developers, decision-makers, and end-users (Ehsan et al., <xref ref-type="bibr" rid="B21">2019</xref>; Samek et al., <xref ref-type="bibr" rid="B68">2019</xref>). For end-users, the requirements and purpose of transparency are expected to be distinct (Cheng et al., <xref ref-type="bibr" rid="B11">2019</xref>; Langer et al., <xref ref-type="bibr" rid="B45">2021</xref>; Suresh et al., <xref ref-type="bibr" rid="B76">2021</xref>), and Miller (<xref ref-type="bibr" rid="B56">2019</xref>) specified criteria that should be taken into account in order to achieve human-centered explainable AI (HCXAI). Following the notion of Ehsan and Riedl (<xref ref-type="bibr" rid="B20">2020</xref>), we understand HCXAI as an approach that puts humans at the center of technology design. Within this framework, not only is it important to conduct user studies that validate XAI methods with ordinary end-users, but also to consider explanations designed to account for human needs. We argue that Miller (<xref ref-type="bibr" rid="B56">2019</xref>)&#x00027;s criteria and the focus on how humans explain decisions to one another are a good starting point for meaningful AI explanations to end-users. However, empirical investigations of the effects of human-centered explanations satisfying these criteria are sparse, and there is mixed evidence about whether transparency is in fact increasing trust (Cramer et al., <xref ref-type="bibr" rid="B13">2008</xref>; Nothdurft et al., <xref ref-type="bibr" rid="B61">2013</xref>; Cheng et al., <xref ref-type="bibr" rid="B11">2019</xref>; Ehsan et al., <xref ref-type="bibr" rid="B21">2019</xref>; Zhang et al., <xref ref-type="bibr" rid="B88">2020</xref>; Poursabzi-Sangdeh et al., <xref ref-type="bibr" rid="B65">2021</xref>). These ambiguous findings may arise from the use of proxy-tasks rather than actual decision-making tasks when evaluating AI systems (Bu&#x000E7;inca et al., <xref ref-type="bibr" rid="B9">2020</xref>) and from varying conceptualizations of trust. Studies on XAI appear to define and measure trust differently (Vereschak et al., <xref ref-type="bibr" rid="B81">2021</xref>). Some researchers assess attitudinal trust measures via questionnaires (Bu&#x000E7;inca et al., <xref ref-type="bibr" rid="B9">2020</xref>), while others focus on trust-related behavior such as reliance (Poursabzi-Sangdeh et al., <xref ref-type="bibr" rid="B65">2021</xref>). However, research has shown that subjective trust can be a poor predictor of actual reliance (Dzindolet et al., <xref ref-type="bibr" rid="B18">2003</xref>; Miller et al., <xref ref-type="bibr" rid="B55">2016</xref>; Papenmeier et al., <xref ref-type="bibr" rid="B62">2022</xref>). Therefore, it seems particularly important to distinguish between attitudinal and behavioral measures when studying the effect of transparency on trust (Parasuraman and Manzey, <xref ref-type="bibr" rid="B63">2010</xref>; Sanneman and Shah, <xref ref-type="bibr" rid="B69">2022</xref>; Scharowski et al., <xref ref-type="bibr" rid="B70">2022</xref>).</p>
<p>In this study, we focus on explainability as a means of AI transparency. Explainability, in this context, is the process of explaining how an opaque box AI arrived at a particular result or decision after a computation has been performed (i.e., <italic>post-hoc</italic> explanations), without directly revealing the AI&#x00027;s internal mechanisms via visualizations or graphical interfaces, as typically aimed for in clear box AI. Grounded in Miller&#x00027;s work, we identified <italic>feature importance explanations</italic> and <italic>counterfactual explanations</italic> as two promising <italic>post-hoc</italic> explanations for achieving HCXAI. We conducted an online decision-making experiment (<italic>N</italic> = 380) on Amazon Mechanical Turk (MTurk) to investigate the effect of those two human-centered explanations on end-users&#x00027; trust and reliance with a control condition. The <italic>Trust between People and Automation Scale</italic> (TPA, Jian et al., <xref ref-type="bibr" rid="B35">2000</xref>) served as an attitudinal measure of AI trust. Reliance on the AI recommendation, captured by <italic>weight of advice</italic>, provided a measure for trust-related behavior. The results suggest that the relationship between transparency and reliance is more nuanced than commonly assumed and emphasize the importance of adequately differentiating between trust and reliance and their respective measurements when evaluating XAI. While transparency did not affect trust, reliance increased through human-centered <italic>post-hoc</italic> explanations, but only for specific decision-making tasks. In the particular context we examined, it appears that the type of decision-making participants were facing (increasing a price vs. decreasing a price) had a greater influence on reliance than how the AI explained its recommendation to the end-users. This suggests that humans display cognitive biases and apply heuristics in decision-making tasks that involve AI recommendations. If biased human decision-making prevails, AI may not support people to reach better decisions. The XAI community should consider potential biases and heuristics for a more nuanced understanding of the human-AI interaction. It remains to be further explored whether measuring attitudinal trust via questionnaires reflects trust-related behavior (i.e., reliance) appropriately and whether heuristics and biases also have an impact on trust. If researchers and practitioners who develop and evaluate AI systems assess only subjective trust, they may not draw valid conclusions about actual AI reliance and vice versa. Given that AI is increasingly utilized to make critical decisions with far-reaching consequences, adopting agreed-upon, validated, and appropriate measurements in XAI is of paramount importance.</p></sec>
<sec id="s2">
<title>2. Related work</title>
<sec>
<title>2.1. Human-centered explanations</title>
<p>Two closely related terms that are often used interchangeably should be distinguished when referring to AI transparency: explainability and interpretability. While both terms refer to methods for achieving transparency, they differ in their approach to implementing transparency. For Lipton (<xref ref-type="bibr" rid="B50">2018b</xref>), interpretability is the information that a system provides about its inner workings and associated with the notion of <italic>clear box</italic> AI, meaning AI whose internal mechanisms are accessible and not concealed. Interpretability is thus achieved by using or designing AI in a way that its decision-making can be directly observed or otherwise visualized. Explainability, on the other hand, implies accepting <italic>opaque box</italic> AI whose internal mechanisms are not readily accessible or understandable, and providing meaningful information by explaining how a specific output or decision was reached after a computation has been carried out. In this sense, explainability is <italic>post-hoc</italic> interpretability (Lipton, <xref ref-type="bibr" rid="B50">2018b</xref>; Ehsan et al., <xref ref-type="bibr" rid="B21">2019</xref>; Miller, <xref ref-type="bibr" rid="B56">2019</xref>; Mohseni et al., <xref ref-type="bibr" rid="B58">2020</xref>).</p>
<p>In addition to this distinction between explainability and interpretability, XAI researchers need to be aware of the varying needs and goals different stakeholders have when interpreting, understanding, and reacting to explanations coming from AI (Suresh et al., <xref ref-type="bibr" rid="B76">2021</xref>). Past research has raised concerns that AI explanations are frequently based on the intuition of researchers, AI developers, and experts rather than addressing the needs of end-users (Du et al., <xref ref-type="bibr" rid="B17">2019</xref>; Miller, <xref ref-type="bibr" rid="B56">2019</xref>). A growing body of work has engaged with this challenge (Ferreira and Monteiro, <xref ref-type="bibr" rid="B24">2020</xref>; Hong et al., <xref ref-type="bibr" rid="B33">2020</xref>; Liao et al., <xref ref-type="bibr" rid="B48">2020</xref>; Ehsan et al., <xref ref-type="bibr" rid="B19">2021a</xref>) and now focuses on more human-centered approaches that align AI explanations with people&#x00027;s needs. Despite these considerations regarding human-centered explanations, previous work on AI transparency has often placed a greater emphasis on interpretability (i.e., model visualization for clear box AI) than on explainability (i.e., <italic>post-hoc</italic> explanations for opaque box AI) (Kulesza et al., <xref ref-type="bibr" rid="B43">2015</xref>; Krause et al., <xref ref-type="bibr" rid="B41">2016</xref>; Cheng et al., <xref ref-type="bibr" rid="B11">2019</xref>; Kocielnik et al., <xref ref-type="bibr" rid="B39">2019</xref>; Lai and Tan, <xref ref-type="bibr" rid="B44">2019</xref>; Poursabzi-Sangdeh et al., <xref ref-type="bibr" rid="B65">2021</xref>). This emphasis has led to focusing on graphical interfaces that allow users to observe and understand the decision-making processes of these models more directly. While <italic>post-hoc</italic> explanations also require some sort of user interface or visualization, they operate at a more abstract level and provide a simplified or approximate representation of the decision-making process rather than direct access to the internal workings of the model, as interpretability seeks to accomplish. However, some researchers have questioned that interpretability approaches are useful to all people equally. Suresh et al. (<xref ref-type="bibr" rid="B76">2021</xref>) and Lipton (<xref ref-type="bibr" rid="B50">2018b</xref>) argue that explainability might be more reflective of the way that humans are transparent about their own decisions. When it comes to humans, the exact processes by which our brains form decisions and our explanations regarding those decisions are distinct (Lipton, <xref ref-type="bibr" rid="B50">2018b</xref>). Similar to how people provide explanations to one another, AI might explain its decisions without disclosing the computation underlying them. Because of its proximity to how humans reason about their decisions, explainability seems promising to achieve HCXAI if the way humans provide and understand explanations is taken into account.</p>
<p>With regard to human-centered explanations, researchers have emphasized the importance of incorporating insights from philosophy, the social sciences, and psychology on how people define, generate, select, evaluate, and present explanations into the field of XAI (Miller, <xref ref-type="bibr" rid="B56">2019</xref>; Mittelstadt et al., <xref ref-type="bibr" rid="B57">2019</xref>). Based on findings from these areas of research, Miller (<xref ref-type="bibr" rid="B56">2019</xref>) defined certain criteria for what contributes to a meaningful explanation for people, including <italic>selectivity</italic> (providing the most important reasons for a decision), <italic>contrastivity</italic> (providing contrastive information with a decision), and <italic>sociality</italic> (explaining something in a similar way to how humans explain their actions). Miller (<xref ref-type="bibr" rid="B56">2019</xref>) and Mittelstadt et al. (<xref ref-type="bibr" rid="B57">2019</xref>) argued that explanations from AI should at least fulfill some of these criteria to be meaningful for end-users. Adadi and Berrada (<xref ref-type="bibr" rid="B3">2018</xref>) identified over 17 different transparency approaches that are being proposed in the current XAI literature. Based on Miller (<xref ref-type="bibr" rid="B56">2019</xref>)&#x00027;s criteria, we narrowed down Adadi and Berrada (<xref ref-type="bibr" rid="B3">2018</xref>)&#x00027;s selection and identified two promising human-centered <italic>post-hoc</italic> explanations: <italic>feature importance explanations</italic> and <italic>counterfactual explanations</italic>.</p>
<p><bold>Feature importance explanations</bold>. Humans rarely expect a complete explanation for a decision and often select the most important or immediate cause from a sometimes infinite number of reasons (Miller, <xref ref-type="bibr" rid="B56">2019</xref>). As the name suggests, feature importance allows end-users to determine which features are most important for an AI&#x00027;s output. Such explanations thus satisfy the selectivity criterion proposed by Miller (<xref ref-type="bibr" rid="B56">2019</xref>) because they show how certain factors influenced a decision. Feature importance explanations have the following notation: &#x0201C;Outcome P was returned because variable V had values (vi, vii, ...) associated with it&#x0201D; (Wachter et al., <xref ref-type="bibr" rid="B82">2018</xref>, p. 9).</p>
<p><bold>Counterfactual explanations</bold>. Humans usually ask why a particular decision was made instead of another one (Miller, <xref ref-type="bibr" rid="B56">2019</xref>). In addition to the leading causes of an output, counterfactuals provide contrastive &#x0201C;what-if&#x0201D; statements that help identify what might be changed in the future to achieve a desired output (Mothilal et al., <xref ref-type="bibr" rid="B59">2020</xref>). Counterfactuals combine Miller&#x00027;s selectivity and contrastivity criteria. They are expected to have psychological benefits because they help people act, rather than merely understand, by altering future behavior to achieve a desired outcome (Wachter et al., <xref ref-type="bibr" rid="B82">2018</xref>; Mothilal et al., <xref ref-type="bibr" rid="B59">2020</xref>). Counterfactuals commonly have the following notation: &#x0201C;Outcome P was returned because variable V had values (vi, vii, ...) associated with it. If V had values (vi&#x00027;, vii&#x00027;, ...) instead, outcome P&#x00027; would have been returned&#x0201D; (Wachter et al., <xref ref-type="bibr" rid="B82">2018</xref>, p.9).</p>
<p>Both explanations also seem to meet Miller (<xref ref-type="bibr" rid="B56">2019</xref>)&#x00027;s sociality criterion. For humans, explanations are a form of social interaction or, more specifically, a transfer of knowledge often presented as part of a conversation between the explainer and the explainee that is subject to the rules of conversation (Hilton, <xref ref-type="bibr" rid="B30">1990</xref>; Miller, <xref ref-type="bibr" rid="B56">2019</xref>). Although Miller (<xref ref-type="bibr" rid="B56">2019</xref>) points out that this does not imply that explanations must be given in natural language, we expect natural language explanations to be a promising approach for human-centered explanations because they are accessible and intuitive to humans (Ehsan et al., <xref ref-type="bibr" rid="B21">2019</xref>). De Graaf and Malle (<xref ref-type="bibr" rid="B15">2018</xref>) argued that because people attribute human-like traits to artificial agents, they might expect them to provide explanations similar to how humans explain their actions. Szymanski et al. (<xref ref-type="bibr" rid="B77">2021</xref>) showed that while end-users prefer visual over textual explanations, they performed significantly worse with the former, and Kizilcec (<xref ref-type="bibr" rid="B37">2016</xref>) demonstrated that short textual explanations build subjective trust in an algorithm&#x00027;s decision. There are also jurisdictional reasons for explanations in natural language. They comply with the EU&#x00027;s GDPR (Wachter et al., <xref ref-type="bibr" rid="B82">2018</xref>) and align with the regulatory requirement for automated decision-making to explain decisions in an &#x0201C;easily accessible form, using clear and plain language [...] provided in writing.&#x0201D; (European Parliament and Council of the European Union, <xref ref-type="bibr" rid="B23">2016</xref>, article 12). To the best of our knowledge, there is little to no empirical research on the effectiveness of these two human-centered explanations derived from the literature (i.e., Adadi and Berrada, <xref ref-type="bibr" rid="B3">2018</xref>) using Miller&#x00027;s criteria in fostering end-users&#x00027; trust in AI. Therefore, an empirical investigation into the efficacy of <italic>feature importance explanations</italic> and <italic>counterfactual explanations</italic> seems warranted.</p></sec>
<sec>
<title>2.2. Trust in XAI</title>
<p>Within the XAI community, researchers define and measure trust in different ways, and there does not appear to be a clear consensus about the desired effect of trust or a clear differentiation of the factors that contribute to trust (Chopra and Wallace, <xref ref-type="bibr" rid="B12">2003</xref>; Mohseni et al., <xref ref-type="bibr" rid="B58">2020</xref>). To provide two examples: Lai and Tan (<xref ref-type="bibr" rid="B44">2019</xref>) proposed a spectrum between full human agency and full automation, with varying levels of explanations along this spectrum. In a deception detection task (asking end-users to decide whether a hotel review is genuine or deceptive), they illustrated that heatmaps of relevant instances and example-based explanations improved human performance and increased the trust humans place on the predictions of the AI. Lai and Tan defined trust as the percentage of instances for which humans relied on the machine prediction. In contrast, Cheng et al. (<xref ref-type="bibr" rid="B11">2019</xref>) conducted an experiment where participants used different UI interfaces to comprehend an algorithm&#x00027;s decision for university admissions. They showed that revealing the inner workings of an algorithm can improve users&#x00027; comprehension and found that users&#x00027; subjective trust, assessed by a 7-point Likert scale, was not affected by the explanation interface. These two empirical studies exemplify how trust is measured differently in XAI research. These discrepancies could be a reason for the inconclusive findings in current XAI literature regarding the effect of transparency on trust. This warrants a more precise definition and rigorous distinction between trust and related concepts, such as reliance, in empirical studies investigating the relationship between transparency and trust.</p>
<p>The differentiation between subjective and objective trust and their measurement in XAI was addressed by Mohseni et al. (<xref ref-type="bibr" rid="B58">2020</xref>). They pointed out that subjective trust measures include interviews, surveys, and self-reports via questionnaires, which according to Bu&#x000E7;inca et al. (<xref ref-type="bibr" rid="B9">2020</xref>) have been the focal points for evaluating AI transparency. For objective measures of trust, Mohseni et al. (<xref ref-type="bibr" rid="B58">2020</xref>) proposed users&#x00027; perceived system competence, understanding, and users&#x00027; reliance on a system. This distinction between trust and reliance was emphasized by Hartmann (<xref ref-type="bibr" rid="B28">2020</xref>). They argued that the everyday use of the word <italic>trust</italic> is misleading when applied to technology and that, in this case, trust must be differentiated from <italic>reliance</italic>. Hartmann (<xref ref-type="bibr" rid="B28">2020</xref>) was not the only one that distinguished between trust and reliance as other researchers have shown that attitudinal judgments have an impact on people&#x00027;s intention to rely on automated systems (Cramer et al., <xref ref-type="bibr" rid="B13">2008</xref>; Merritt, <xref ref-type="bibr" rid="B54">2011</xref>); People tend to rely on automation they trust and reject automation they distrust (Lee and See, <xref ref-type="bibr" rid="B47">2004</xref>). This makes trust particularly relevant in the misuse (overreliance Parasuraman and Riley, <xref ref-type="bibr" rid="B64">1997</xref>) and disuse (neglect or underutilization Parasuraman and Riley, <xref ref-type="bibr" rid="B64">1997</xref>) of automation (Hoff and Bashir, <xref ref-type="bibr" rid="B31">2015</xref>; Yu et al., <xref ref-type="bibr" rid="B87">2017</xref>; Stephanidis et al., <xref ref-type="bibr" rid="B75">2019</xref>). To avoid such instances, users&#x00027; trust needs to be calibrated or warranted. Trust calibration refers to the extent to which the trust that users place in the system is adequate to the system&#x00027;s actual capabilities (Wischnewski et al., <xref ref-type="bibr" rid="B85">2023</xref>). Fostering end users&#x00027; trust in AI should aim to attain an appropriate level of trust to avoid overreliance or underutilization of AI systems. According to Lee and See (<xref ref-type="bibr" rid="B47">2004</xref>) and correspondent with Hoff and Bashir (<xref ref-type="bibr" rid="B31">2015</xref>), we define trust in the context of AI as &#x0201C;the attitude that an agent will help achieve an individual&#x00027;s goals in a situation characterized by uncertainty and vulnerability&#x0201D; (Lee and See, <xref ref-type="bibr" rid="B47">2004</xref>, p. 6). This definition encapsulates the notion of uncertainty and vulnerability as proposed by Jacovi et al. (<xref ref-type="bibr" rid="B34">2021</xref>) and Mayer et al. (<xref ref-type="bibr" rid="B53">1995</xref>), which is the most widely used and accepted definition of trust (Rousseau et al., <xref ref-type="bibr" rid="B67">1998</xref>). Adopting Lee and See&#x00027;s definition and model, we distinguish between trust and reliance and think of trust as an <italic>attitude</italic> and reliance as a <italic>behavior</italic> that follows the level of trust. In their work on metrics for XAI, Hoffman et al. (<xref ref-type="bibr" rid="B32">2019</xref>) make a similar distinction when they differentiate between trusting a machine&#x00027;s output and following its advice. In this framework, attitudes and behaviors remain conceptually distinct and do not share a deterministic but a probabilistic relationship (Ajzen and Fishbein, <xref ref-type="bibr" rid="B4">1980</xref>; K&#x000F6;rber, <xref ref-type="bibr" rid="B40">2018</xref>). Even if an AI system is trusted, reliance must not necessarily follow (Kirlik, <xref ref-type="bibr" rid="B36">1993</xref>; K&#x000F6;rber, <xref ref-type="bibr" rid="B40">2018</xref>), and people may claim to trust an AI system, yet behave in a way that suggests they do not (Miller et al., <xref ref-type="bibr" rid="B55">2016</xref>). This implies that attitudes may not always translate into behaviors. The empirical findings of Dzindolet et al. (<xref ref-type="bibr" rid="B18">2003</xref>) support this argument: although some automated decision aids were rated as more trustworthy than others, all were equally likely to be relied upon.</p>
<p>Given these possible contradictions, we think it is useful to conceptualize trust as an antecedent to reliance that guides but does not determine it (Lee and See, <xref ref-type="bibr" rid="B47">2004</xref>). However, this consideration has been insufficiently taken into account in past research (Papenmeier et al., <xref ref-type="bibr" rid="B62">2022</xref>; Scharowski et al., <xref ref-type="bibr" rid="B70">2022</xref>). A rigorous distinction and an accurate conceptualization of trust and reliance are vital for empirical XAI studies since researchers who evaluate AI systems using only subjective measures of AI trust might not draw valid conclusions about actual reliance on AI and vice versa. Arrieta et al. (<xref ref-type="bibr" rid="B6">2020</xref>) emphasized that only agreed-upon metrics and their respective measurements allow for meaningful comparisons in XAI research and that without such consistency, any claims are elusive and do not provide a solid foundation. For this reason, we decided to investigate two alternative methodological approaches, namely, measuring attitudinal trust on the one hand and measuring trust-related behavior in terms of reliance on the other hand. This approach is in line with Sanneman and Shah (<xref ref-type="bibr" rid="B69">2022</xref>), that recommended using trust scales in conjunction with behavior-based metrics to determine if people appropriately trust <italic>and</italic> use AI systems in response to AI explanations they provide.</p></sec></sec>
<sec id="s3">
<title>3. Empirical investigation</title>
<sec>
<title>3.1. Research question and hypotheses</title>
<p>We investigated the following research question:</p>
<p><bold>RQ: What effect do human-centered explanations have on end-users&#x00027; trust and reliance?</bold></p>
<p>To answer this research question, we compared the previously introduced feature importance and counterfactual <italic>post-hoc</italic> explanations with a control in a scenario in which participants had to estimate subleasing prices for different apartments. We employed a mixed study design with a 3 (explanation condition: feature importance vs. counterfactual vs. control) &#x000D7; 2 (type of AI recommendation: increasing price vs. decreasing price). Explanation condition was the between-subject factor, type of recommendation was the within-subject factor. Following Poursabzi-Sangdeh et al. (<xref ref-type="bibr" rid="B65">2021</xref>), we focused on the domain of real estate valuation, where machine learning is often used to predict apartment prices. Airbnb (<ext-link ext-link-type="uri" xlink:href="https://airbnb.com">https://airbnb.com</ext-link>) and Zillow (<ext-link ext-link-type="uri" xlink:href="http://zillow.com">http://zillow.com</ext-link>) are examples of websites that provide price recommendations to end-users in this way. Considering the previous clarifications, we expected that trust and reliance are influenced by human-centered explanations similarly but should be treated as distinct concepts. We, therefore, formulated separate hypotheses for both trust and reliance. We further presumed that feature importance and counterfactual explanations lead to more trust <italic>and</italic> reliance in participants compared to a control condition where no additional explanation was present. Counterfactuals are both selective and contrastive, while feature importance explanations are just selective (Miller, <xref ref-type="bibr" rid="B56">2019</xref>). This makes counterfactuals an even more promising type of human-centered explanation compared to feature importance explanations.</p>
<p>For these reasons, the specific hypotheses were:</p>
<list list-type="simple">
<list-item><p><bold>H</bold><sub><bold>1</bold></sub> The experimental condition <italic>feature importance</italic> will lead to higher reliance compared to the control.</p></list-item>
<list-item><p><bold>H</bold><sub><bold>2</bold></sub>: The experimental condition <italic>counterfactuals</italic> will lead to higher reliance compared to the control.</p></list-item>
<list-item><p><bold>H</bold><sub><bold>3</bold></sub>: <italic>Counterfactuals</italic> will lead to higher reliance compared to <italic>feature importance</italic>.</p></list-item>
<list-item><p><bold>H</bold><sub><bold>4</bold></sub>: The experimental condition <italic>feature importance</italic> will lead to higher trust compared to the control.</p></list-item>
<list-item><p><bold>H</bold><sub><bold>5</bold></sub>: The experimental condition <italic>counterfactuals</italic> will lead to higher trust compared to the control.</p></list-item>
<list-item><p><bold>H</bold><sub><bold>6</bold></sub>: <italic>Counterfactuals</italic> will lead to higher trust compared to <italic>feature importance</italic>.</p></list-item>
</list></sec></sec>
<sec id="s4">
<title>4. Method</title>
<sec>
<title>4.1. Measures</title>
<p>The independent variable was <italic>condition</italic>, with the two levels feature importance and counterfactuals, as well as a third level without explanations, which served as a control. We used two measures as dependent variables to account for the aforementioned distinction between trust as an attitude and reliance as trust-related behavior. On the one hand, we wanted to determine if people relied on the AI and changed their behavior after being presented with an explanation. This behavior change was captured by the parameter <italic>Weight of Advice</italic> (WOA), which stems from the literature on taking advice (Harvey and Fischer, <xref ref-type="bibr" rid="B29">1997</xref>). WOA has the following notation:</p>
<disp-formula id="E1"><label>(1)</label><mml:math id="M1"><mml:mtable class="eqnarray" columnalign="left"><mml:mtr><mml:mtd><mml:mstyle mathvariant='bold'><mml:mtext>WOA</mml:mtext></mml:mstyle><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:mi>T</mml:mi><mml:mn>2</mml:mn><mml:mo>-</mml:mo><mml:mi>T</mml:mi><mml:mn>1</mml:mn></mml:mrow><mml:mrow><mml:mi>R</mml:mi><mml:mo>-</mml:mo><mml:mi>T</mml:mi><mml:mn>1</mml:mn></mml:mrow></mml:mfrac></mml:mtd></mml:mtr></mml:mtable></mml:math></disp-formula>
<p>In Equation (1), <italic>R</italic> is defined as the model&#x00027;s recommendation, <italic>T</italic>1 is the participant&#x00027;s initial estimate of the apartment&#x00027;s price before seeing <italic>R</italic>, and <italic>T</italic>2 is the participant&#x00027;s final estimate of the apartment&#x00027;s price after seeing <italic>R</italic>. WOA measures the degree to which people change their behavior and move their initial estimate toward the advice. WOA is equal to 1 if the participant&#x00027;s final prediction matches the AI recommendation and equal to 0.5 if they average their initial prediction with the AI recommendation. A WOA of 0 occurs when a participant ignores the AI recommendation (<italic>T</italic>1 &#x0003D; <italic>T</italic>2), and a negative WOA signifies that a participant discounted the recommendation completely and moved further away from the recommendation. WOA can be viewed as a percentage of how much people weigh the received advice (i.e., the AI recommendation), and this straightforward interpretation is an advantage of this reliance measurement. While WOA has been used in the past by researchers in XAI as an alternative trust measurement (Logg et al., <xref ref-type="bibr" rid="B51">2019</xref>; Mucha et al., <xref ref-type="bibr" rid="B60">2021</xref>; Poursabzi-Sangdeh et al., <xref ref-type="bibr" rid="B65">2021</xref>), it has never been explicitly referred to as reliance and thus clearly differentiated from trust to the best of our knowledge.</p>
<p>On the other hand, we chose the TPA (Jian et al., <xref ref-type="bibr" rid="B35">2000</xref>) to measure trust because the scale&#x00027;s underlying definition of trust is compatible with the one we adopted from Lee and See (<xref ref-type="bibr" rid="B47">2004</xref>). Furthermore, the TPA is an established measure in HCI (Hoffman et al., <xref ref-type="bibr" rid="B32">2019</xref>). Several other scales evaluating AI have adapted items from the TPA (e.g., Hoffman et al., <xref ref-type="bibr" rid="B32">2019</xref>), and its psychometric quality has been evaluated multiple times (Spain et al., <xref ref-type="bibr" rid="B74">2008</xref>; Gutzwiller et al., <xref ref-type="bibr" rid="B27">2019</xref>). Jian et al. (<xref ref-type="bibr" rid="B35">2000</xref>) treated trust and distrust as opposite factors extending along a single dimension of trust. The scale is a seven-point Likert-type scale (ranging from 1: &#x0201C;not at all&#x0201D; to 7: &#x0201C;extremely&#x0201D;) and consists of 12 items. Five items for distrust (i.e., &#x0201C;<italic>The system is deceptive.&#x0201D;, &#x0201C;The system behaves in an underhanded manner.&#x0201D;, &#x0201C;I am suspicious of the system&#x00027;s intent, action or, outputs.&#x0201D;, &#x0201C;I am wary of the system.&#x0201D;, &#x0201C;The system&#x00027;s actions will have a harmful or injurious outcome.&#x0201D;</italic>). The seven remaining items for trust (i.e., &#x0201C;<italic>I am confident in the system.&#x0201D;, &#x0201C;The system provides security.&#x0201D;, &#x0201C;The system has integrity.&#x0201D;, &#x0201C;The system is dependable.&#x0201D;, &#x0201C;The system is reliable.&#x0201D;, &#x0201C;I can trust the system.&#x0201D;, &#x0201C;I am familiar with the system.&#x0201D;</italic>). We used the scale in its original form, except for prefixing the word &#x0201C;AI&#x0201D; to the word &#x0201C;system,&#x0201D; e.g., &#x0201C;I have confidence in the AI system.&#x0201D;</p></sec>
<sec>
<title>4.2. Experiment</title>
<p>We carried out a one-factor between-subjects design online experiment<xref ref-type="fn" rid="fn0001"><sup>1</sup></xref> over Amazon Mechanical Turk (MTurk, <ext-link ext-link-type="uri" xlink:href="http://mturk.com">http://mturk.com</ext-link>). The experiment was implemented through the online survey tool Limesurvey (<ext-link ext-link-type="uri" xlink:href="http://limesurvey.org">http://limesurvey.org</ext-link>).</p>
<sec>
<title>4.2.1. Participants</title>
<p>A total of 913 participants were initially recruited over MTurk, and 798 of them fully completed the survey. Only workers from the USA with a human-intelligence-task (HIT) approval of 95% and at least 100 approved HITs were allowed to participate in the experiment. Workers who completed the task conscientiously were reimbursed with 1.50 US dollars and a bonus of 0.30 US dollars for their participation. Several criteria were applied during data cleaning to ensure data quality. Participants who failed to provide a correct answer (<italic>n</italic> &#x0003D; 36) for the bogus item (&#x0201C;This is an attention check. Please choose 7 here&#x0201D;) or for one of three control questions (&#x0201C;In this survey, you had to tell us for how much money you would sell a house to a company&#x0201D;; &#x0201C;In this survey, we asked you to indicate in which U.S. state you currently live&#x0201D;; &#x0201C;In this survey, you got price recommendations from a good friend&#x0201D;) were removed (<italic>n</italic> &#x0003D; 310). We also excluded participants that showed unrealistic WOA&#x00027;s (<italic>n</italic> &#x0003D; 72). Following prior research (Gino and Moore, <xref ref-type="bibr" rid="B26">2007</xref>; Logg et al., <xref ref-type="bibr" rid="B51">2019</xref>), we defined unrealistic WOA as being &#x02264; &#x02212;1 and &#x02265;2. For the data analysis, 380 participants remained. The sample was predominantly male (61%) and had an average age of 37 years (<italic>M</italic> &#x0003D; 37.03, <italic>SD</italic> &#x0003D; 10.15, <italic>min</italic> &#x0003D; 18, <italic>max</italic> &#x0003D; 69). A majority of the participants (68%) possessed a higher-educational degree (i.e., a bachelor&#x00027;s degree, master&#x00027;s degree, or PhD).</p></sec>
<sec>
<title>4.2.2. Procedure and task</title>
<p>After providing informed consent, participants were introduced to the study and their task. They were asked to imagine a scenario where their goal was to sublease six different apartments on a subleasing website. Based on the apartment&#x00027;s features and amenities (e.g., number of bedrooms, distance to public transit), they had to estimate an initial subleasing price (<italic>T</italic>1). After estimating <italic>T</italic>1, an alleged AI from the website provided a computed price recommendation (<italic>R</italic>). In reality, the price recommendation was based on an algorithm introduced as an AI. Participants were informed that they would be more likely to find a subleaser by deciding on a lower price but would consequently receive less profit. If they decided on a higher price, they would be less likely to find a subleaser but potentially receive more profit. They were told that the AI&#x00027;s goal was to help them find the optimal price to successfully find a subleaser with a reasonable profit. How exactly this price recommendation was calculated by the algorithm will be discussed in the next section. <xref ref-type="fig" rid="F1">Figure 1</xref> shows how the price recommendation and respective explanations were presented to the participants. A list of all the explanations used for each type of recommendation can be found in the online repository<xref ref-type="fn" rid="fn0001"><sup>1</sup></xref>. The output was designed to appear as if the subsequent explanation was an extension of the preceding ones. The most relevant outputs were presented as a console output in order to make the stimuli more convincing (see <xref ref-type="fig" rid="F1">Figure 1</xref>). After seeing the AI recommendation, participants could decide if they desired to approach it or not and settled on a final subleasing price (<italic>T</italic>2). This deliberate choice by the participants to either rely on the AI recommendation or not makes our experiment an actual decision-making task rather than a proxy task (Bu&#x000E7;inca et al., <xref ref-type="bibr" rid="B9">2020</xref>). In proxy-tasks, the focus lies on how well participants can simulate the model&#x00027;s decisions (Bu&#x000E7;inca et al., <xref ref-type="bibr" rid="B9">2020</xref>). In actual decision-making tasks, people&#x00027;s choices involve systematic thinking errors (biases) and mental shortcuts (heuristics) (Tversky and Kahneman, <xref ref-type="bibr" rid="B78">1974</xref>), as it is up to the participants to decide whether and how to use the AI (i.e., reliance on the AI).</p>
<fig id="F1" position="float">
<label>Figure 1</label>
<caption><p>Examples how the different explanations for the conditions <bold>(A)</bold> Control, <bold>(B)</bold> Feature Importance, and <bold>(C)</bold> Counterfactual were presented to the participants. For this apartment, the AI recommended increasing the initial price <italic>T</italic>1.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fcomp-05-1151150-g0001.tif"/>
</fig>
<p>To evoke a certain degree of uncertainty, participants were told that they would be reimbursed based on their performance. Uncertainty is a defining characteristic of trust (Mayer et al., <xref ref-type="bibr" rid="B53">1995</xref>) and has been referred to as a necessary prerequisite of human trust in AI that has been lacking in current XAI studies (Jacovi et al., <xref ref-type="bibr" rid="B34">2021</xref>). Participants were informed that for good estimations, the top 10% would be paid an additional bonus of 0.30 US dollars. In actuality, every participant received the bonus, regardless of their performance. In order to better control for the price disparity between urban and rural regions, participants were asked to indicate what US state they are currently living in (e.g., Colorado) to ascertain their state capital (e.g., Denver). The objective was to make the estimate easier for the participants and to make the AI more persuasive since it was claimed that the AI would likewise base its price recommendations on data collected in that state capital. After an example that showed how the apartments and their amenities would be presented to them, participants could start with the actual task. Once the task was completed, participants had to fill out the TPA (Jian et al., <xref ref-type="bibr" rid="B35">2000</xref>) and give some demographic information. Participants were debriefed at the end of the study and informed that the AI was an algorithm introduced as an AI that did not use participants&#x00027; state capitals for its recommendations. To ascertain that our algorithm was convincing, we asked participants before and after the interaction how certain they were about the AI&#x00027;s prediction. (&#x0201C;How certain are you that the AI can make an accurate recommendation for a sublease price?&#x0201D;) on a ten-point Likert-type scale (ranging from 1: &#x0201C;not certain at all&#x0201D; to 10: &#x0201C;absolutely certain&#x0201D;).</p></sec>
<sec>
<title>4.2.3. Stimuli</title>
<p>The apartments that participants had to evaluate were real apartments retrieved from the website Zillow (<ext-link ext-link-type="uri" xlink:href="http://zillow.com">http://zillow.com</ext-link>) in May 2020. To create some variability, we selected six different apartments of different sizes and price ranges: two small-sized apartments (500&#x02013;750 square feet), two medium-sized apartments (751&#x02013;1,000 square feet), and two large-sized apartments (1,001&#x02013;1,250 square feet). <xref ref-type="fig" rid="F2">Figure 2</xref> shows an example of how apartments were presented to participants. Features and amenities were collected directly from the website Zillow, whenever available. If not available, a random value within a reasonable range was chosen for continuous variables (e.g., distance to public transit between 0.1 and 2.0 miles), and a random choice for dichotomous variables was made (e.g., elevator YES/NO). All participants were presented with the same stimuli, that is, identical apartments and features. What differed was the price recommendation and the explanation that accompanied it (see <xref ref-type="fig" rid="F1">Figure 1</xref>).</p>
<fig id="F2" position="float">
<label>Figure 2</label>
<caption><p>Example of how an apartment was presented to participants in the online experiment. Note that the image of the apartment depicted in the screenshot was replaced with a similar image for this publication due to potential copyright issues.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fcomp-05-1151150-g0002.tif"/>
</fig>
<p>The price recommendation from the algorithm introduced as an AI was designed to pick a random number between 10 and 20. This random number was then transformed into percentages and either added or subtracted to the initial subleasing price (<italic>T</italic>1), which led to a random deviation between 10 and 20 percent. This deviation seemed substantial enough that subjects did not entirely adopt the recommendation, but it was also subtle enough not to appear unrealistic and that it seemed possible that the features and amenities could account for the discrepancy. With this procedure, we ensured that no participant could estimate the price accurately since there was no &#x0201C;true price.&#x0201D; Defining a ground truth has been a limitation of past studies (Poursabzi-Sangdeh et al., <xref ref-type="bibr" rid="B65">2021</xref>). If, by pure chance, a participant estimates the &#x0201C;true price,&#x0201D; the interpretation of WOA becomes meaningless since <italic>T</italic>1 and <italic>R</italic> are equal. The absence of a &#x0201C;true price&#x0201D; imposes the decision on participants either to rely or not to rely on the AI. By determining a relative deviation between 10 and 20 percent from participants&#x00027; initial price estimate, we furthermore controlled for the system&#x00027;s accuracy since it was shown to have a significant effect on people&#x00027;s trust (Yin et al., <xref ref-type="bibr" rid="B86">2019</xref>; Zhang et al., <xref ref-type="bibr" rid="B88">2020</xref>).</p>
<p>It was randomly assigned that for three of the six apartments, the algorithm introduced as an AI recommended decreasing the initial price, <italic>T</italic>1 (e.g., if <italic>T</italic>1 was $1,000 and the random number 17, the AI recommendation was $830), and for the other three apartments, the recommendation was to increase <italic>T</italic>1 (e.g., if <italic>T</italic>1 was $1,000 and the random number 17, the AI recommendation was $1,170). By doing this, the AI informed participants that their initial price estimates were either too low or too high, which made it possible to compare AI recommendations to <italic>increase</italic> <italic>T</italic>1 with recommendations to <italic>decrease</italic> <italic>T</italic>1. We were interested in this comparison because prior research from Kliegr et al. (<xref ref-type="bibr" rid="B38">2021</xref>) and Wang et al. (<xref ref-type="bibr" rid="B83">2019</xref>) led us to postulate that the AI&#x00027;s recommendation to increase or decrease the initial price might also influence participants&#x00027; decision-making and consequently their reliance on the AI.</p></sec></sec></sec>
<sec sec-type="results" id="s5">
<title>5. Results</title>
<sec>
<title>5.1. Descriptive statistics</title>
<p>On average, participants across all conditions approached the AI recommendation, resulting in a positive WOA (<italic>M</italic> &#x0003D; 0.69, <italic>SD</italic> &#x0003D; 0.36). The TPA showed average overall ratings (<italic>M</italic> &#x0003D; 5.01, <italic>SD</italic> &#x0003D; 0.86), high ratings for trust (<italic>M</italic> &#x0003D; 4.98, <italic>SD</italic> &#x0003D; 1.05), and lower ratings for distrust (<italic>M</italic> &#x0003D; 2.95, <italic>SD</italic> &#x0003D; 1.62). Across all three conditions, the certainty that the AI could make an accurate prediction increased from pre- to post-interaction (<italic>M</italic><sub>&#x00394;</sub> &#x0003D; 0.36, <italic>SD</italic> &#x0003D; 1.53) and was rated at a high level after the interaction (<italic>M</italic> &#x0003D; 7.59, <italic>SD</italic> &#x0003D; 1.60). The inspection of the average estimated prices also confirmed our classifications into the apartment categories &#x0201C;small,&#x0201D; &#x0201C;medium,&#x0201D; and &#x0201C;large&#x0201D; (<italic>M</italic><sub><italic>small</italic></sub> &#x0003D; <italic>$</italic>1, 091, <italic>M</italic><sub><italic>medium</italic></sub> &#x0003D; <italic>$</italic>1, 286, <italic>M</italic><sub><italic>large</italic></sub> &#x0003D; <italic>$</italic>1, 449). From this, we concluded that the assigned task was a compelling one. <xref ref-type="table" rid="T1">Table 1</xref> includes descriptive statistics for the experiment.</p>
<table-wrap position="float" id="T1">
<label>Table 1</label>
<caption><p>Descriptive statistics of the conducted experiment with the mean (<italic>M</italic>), standard deviation (<italic>SD</italic>), and median (<italic>Mdn</italic>) for <italic>WOA</italic>, the TPA, and <italic>AI</italic> certainty.</p></caption> 
<table frame="box" rules="all">
<thead>
<tr style="background-color:&#x00023;919498;color:&#x00023;ffffff">
<th/>
<th valign="top" align="center" colspan="3"><bold>Control</bold></th>
<th valign="top" align="center" colspan="3"><bold>Feature importance</bold></th>
<th valign="top" align="center" colspan="3"><bold>Counterfactual</bold></th>
</tr>
<tr style="background-color:&#x00023;919498;color:&#x00023;ffffff">
<td/>
<td valign="top" align="center" colspan="3"><bold>(</bold><italic><bold>n</bold></italic> <bold>= 130)</bold></td>
<td valign="top" align="center" colspan="3"><bold>(</bold><italic><bold>n</bold></italic> <bold>= 143)</bold></td>
<td valign="top" align="center" colspan="3"><bold>(</bold><italic><bold>n</bold></italic> <bold>= 107)</bold></td>
</tr>
 <tr style="background-color:&#x00023;919498;color:&#x00023;ffffff">
<td/>
<td valign="top" align="center"><italic><bold>M</bold></italic></td>
<td valign="top" align="center"><italic><bold>SD</bold></italic></td>
<td valign="top" align="center"><italic><bold>Mdn</bold></italic></td>
<td valign="top" align="center"><italic><bold>M</bold></italic></td>
<td valign="top" align="center"><italic><bold>SD</bold></italic></td>
<td valign="top" align="center"><italic><bold>Mdn</bold></italic></td>
<td valign="top" align="center"><italic><bold>M</bold></italic></td>
<td valign="top" align="center"><italic><bold>SD</bold></italic></td>
<td valign="top" align="center"><italic><bold>Mdn</bold></italic></td>
</tr> 
</thead>
<tbody>
<tr style="background-color:#dee1e1">
<td valign="top" align="left" colspan="10"><bold>Weight of Advice</bold></td>
</tr> <tr>
<td valign="top" align="left"><italic>Increase price</italic></td>
<td valign="top" align="center">0.69</td>
<td valign="top" align="center">0.38</td>
<td valign="top" align="center">0.69</td>
<td valign="top" align="center">0.68</td>
<td valign="top" align="center">0.36</td>
<td valign="top" align="center">0.69</td>
<td valign="top" align="center">0.67</td>
<td valign="top" align="center">0.34</td>
<td valign="top" align="center">0.67</td>
</tr> <tr>
<td valign="top" align="left"><italic>Decrease price</italic></td>
<td valign="top" align="center">0.69</td>
<td valign="top" align="center">0.38</td>
<td valign="top" align="center">0.69</td>
<td valign="top" align="center">0.73</td>
<td valign="top" align="center">0.40</td>
<td valign="top" align="center">0.79</td>
<td valign="top" align="center">0.78</td>
<td valign="top" align="center">0.37</td>
<td valign="top" align="center">0.79</td>
</tr> <tr style="background-color:#dee1e1">
<td valign="top" align="left" colspan="10"><bold>TPA (Jian et al.</bold>, <xref ref-type="bibr" rid="B35"><bold>2000</bold></xref><bold>)</bold></td>
</tr> <tr>
<td valign="top" align="left"><italic>Overall</italic></td>
<td valign="top" align="center">5.01</td>
<td valign="top" align="center">0.91</td>
<td valign="top" align="center">4.88</td>
<td valign="top" align="center">4.95</td>
<td valign="top" align="center">0.84</td>
<td valign="top" align="center">4.83</td>
<td valign="top" align="center">5.10</td>
<td valign="top" align="center">0.84</td>
<td valign="top" align="center">5.08</td>
</tr> <tr>
<td valign="top" align="left"><italic>Trust</italic></td>
<td valign="top" align="center">5.11</td>
<td valign="top" align="center">1.06</td>
<td valign="top" align="center">5.29</td>
<td valign="top" align="center">4.82</td>
<td valign="top" align="center">1.07</td>
<td valign="top" align="center">5.00</td>
<td valign="top" align="center">5.03</td>
<td valign="top" align="center">1.00</td>
<td valign="top" align="center">5.14</td>
</tr> <tr>
<td valign="top" align="left"><italic>Distrust</italic></td>
<td valign="top" align="center">3.12</td>
<td valign="top" align="center">1.73</td>
<td valign="top" align="center">2.60</td>
<td valign="top" align="center">2.88</td>
<td valign="top" align="center">1.58</td>
<td valign="top" align="center">2.40</td>
<td valign="top" align="center">2.82</td>
<td valign="top" align="center">1.54</td>
<td valign="top" align="center">2.20</td>
</tr> <tr style="background-color:#dee1e1">
<td valign="top" align="left" colspan="10"><bold>AI certainty</bold></td>
</tr> <tr>
<td valign="top" align="left"><italic>Pre-interaction</italic></td>
<td valign="top" align="center">7.24</td>
<td valign="top" align="center">1.60</td>
<td valign="top" align="center">8.00</td>
<td valign="top" align="center">7.20</td>
<td valign="top" align="center">1.57</td>
<td valign="top" align="center">8.00</td>
<td valign="top" align="center">7.27</td>
<td valign="top" align="center">1.64</td>
<td valign="top" align="center">8.00</td>
</tr> <tr>
<td valign="top" align="left"><italic>Post-interaction</italic></td>
<td valign="top" align="center">7.55</td>
<td valign="top" align="center">1.74</td>
<td valign="top" align="center">8.00</td>
<td valign="top" align="center">7.48</td>
<td valign="top" align="center">1.61</td>
<td valign="top" align="center">8.00</td>
<td valign="top" align="center">7.80</td>
<td valign="top" align="center">1.37</td>
<td valign="top" align="center">8.00</td>
</tr></tbody>
</table>
</table-wrap></sec>
<sec>
<title>5.2. Reliance&#x02014;WOA</title>
<p>To address <bold>H</bold><sub><bold>1</bold></sub>, <bold>H</bold><sub><bold>2</bold></sub>, and <bold>H</bold><sub><bold>3</bold></sub>, corresponding contrasts were created. The first contrast made it possible to determine if the feature importance condition was significantly different from the control (planned contrast 1: feature importance explanation vs. control for answering <bold>H</bold><sub><bold>1</bold></sub>). By defining two other contrasts, it was possible to examine if the counterfactual condition was significantly different from the control (planned contrast 2: counterfactual explanation vs. control for answering <bold>H</bold><sub><bold>2</bold></sub>) and if the counterfactual condition was significantly different from the feature importance condition (planned contrast 3: counterfactual explanation vs. feature importance explanation for answering <bold>H</bold><sub><bold>3</bold></sub>). The effect of the three contrasts on WOA was analyzed by employing linear mixed-effect models (LMEMs) using the <italic>lme4</italic> package (Bates et al., <xref ref-type="bibr" rid="B7">2015</xref>) for R (version 4.2.2.). We report <italic>&#x003B2;</italic>-estimates, their 95% confidence interval, <italic>t</italic>-values, and the corresponding <italic>p</italic>-values. Our models contained two fixed effects: the contrasts and the difference of the recommendation to <italic>increase</italic> or <italic>decrease</italic> <italic>T</italic>1. Under the assumption that the stimuli and conditions had varying random effects for different participants, we introduced a random intercept (id) in the model. The utilized model had the following specifications:</p>
<disp-formula id="E2"><mml:math id="M2"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mi>W</mml:mi><mml:mi>O</mml:mi><mml:mi>A</mml:mi><mml:mo>&#x0007E;</mml:mo><mml:mn>1</mml:mn><mml:mo>&#x0002B;</mml:mo><mml:mi>C</mml:mi><mml:mi>o</mml:mi><mml:mi>n</mml:mi><mml:mi>t</mml:mi><mml:mi>r</mml:mi><mml:mi>a</mml:mi><mml:mi>s</mml:mi><mml:mi>t</mml:mi><mml:mn>1</mml:mn><mml:mo>&#x0002B;</mml:mo><mml:mi>R</mml:mi><mml:mi>e</mml:mi><mml:mi>c</mml:mi><mml:mi>o</mml:mi><mml:mi>m</mml:mi><mml:mi>m</mml:mi><mml:mi>e</mml:mi><mml:mi>n</mml:mi><mml:mi>d</mml:mi><mml:mi>a</mml:mi><mml:mi>t</mml:mi><mml:mi>i</mml:mi><mml:mi>o</mml:mi><mml:mi>n</mml:mi><mml:mo>&#x0002B;</mml:mo><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:mn>1</mml:mn><mml:mo>|</mml:mo><mml:mi>i</mml:mi><mml:mi>d</mml:mi></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow></mml:mtd></mml:mtr></mml:mtable></mml:math></disp-formula>
<p>For this model, the first contrast (feature importance explanation vs. control) was not significant [<italic>&#x003B2;</italic> &#x0003D; 0.01, 95%&#x02002;<italic>CI</italic>&#x02002;<italic>&#x003B2;</italic>[-0.01, 0.03], <italic>t</italic><sub>(378)</sub> &#x0003D; 0.64, <italic>p</italic> &#x0003D; 0.53], while the difference between recommendations to increase or decrease <italic>T</italic>1 was highly significant [<italic>&#x003B2;</italic> &#x0003D; 0.05, 95%&#x02002;<italic>CI</italic>&#x02002;<italic>&#x003B2;</italic>[0.03, 0.07], <italic>t</italic><sub>(1,899)</sub> &#x0003D; 3.62, <italic>p</italic> &#x0003C; 0.001]. The second contrast (counterfactual explanation vs. control) did not return any significant results [<italic>&#x003B2;</italic> &#x0003D; 0.02, 95%&#x02002;<italic>CI</italic>&#x02002;<italic>&#x003B2;</italic>[-0.01, 0.04], <italic>t</italic><sub>(378)</sub> &#x0003D; 1.33, <italic>p</italic> &#x0003D; 0.19], and neither did the third contrast (counterfactual explanation vs. feature importance explanation) [<italic>&#x003B2;</italic> &#x0003D; 0.01, 95%&#x02002;<italic>CI</italic>&#x02002;<italic>&#x003B2;</italic>[-0.01, 0.03], <italic>t</italic><sub>(378)</sub> &#x0003D; 0.63, <italic>p</italic> &#x0003D; 0.53]. However, in all three models, the recommendation type (increasing a price vs. decreasing a price) returned highly significant results with <italic>&#x003B2;</italic>-estimates ranging between 95%&#x02002;<italic>CI</italic>&#x02002;[0.03, 0.07]. Comparing this model to a model without the recommendation term confirmed that its inclusion was justified since it significantly improved the model fit [<inline-formula><mml:math id="M3"><mml:msubsup><mml:mrow><mml:mi>&#x003C7;</mml:mi></mml:mrow><mml:mrow><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:mn>1</mml:mn></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow></mml:mrow><mml:mrow><mml:mn>2</mml:mn></mml:mrow></mml:msubsup><mml:mo>=</mml:mo><mml:mn>13</mml:mn><mml:mo>.</mml:mo><mml:mn>05</mml:mn></mml:math></inline-formula>, <italic>p</italic> &#x0003C; 0.001]. To better understand the relationship between explanations and the type of recommendation, we created a visualization (see <xref ref-type="fig" rid="F3">Figure 3</xref>).</p>
<fig id="F3" position="float">
<label>Figure 3</label>
<caption><p>Mean weight of advice for each condition and type of recommendation. Note that the y-axis is scaled to better visualize the effect. The error bars depict 95% confidence intervals. *Statistically significant difference with <italic>p</italic> &#x0003C; 0.05.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fcomp-05-1151150-g0003.tif"/>
</fig>
<p>Depending on the type of recommendation, the condition effect was different, meaning that whether the AI recommended <italic>increasing</italic> or <italic>decreasing</italic> the initial subleasing price <italic>T</italic>1, influenced the way that explanations affected WOA. For recommendations to increase, the explanations had a negligible effect on WOA, but for recommendations to decrease, the effect was substantial. In our case, the effect of explanations cannot be readily understood without considering the different type of AI recommendation. We therefore divided the data into two subsets. One subset contained the three apartments with the <italic>recommendation to increase</italic> <italic>T</italic>1, the other subset contained the three apartments with the <italic>recommendation to decrease</italic> <italic>T</italic>1. We then executed the specified model again, but the term &#x0201C;recommendation&#x0201D; was naturally omitted as a fixed effect. For the subset that contained the recommendations to decrease <italic>T</italic>1, the second contrast (counterfactual explanation vs. control) was significant [<italic>&#x003B2;</italic> &#x0003D; 0.04, 95%&#x02002;<italic>CI</italic>&#x02002;<italic>&#x003B2;</italic>[0.01, 0.08], <italic>t</italic><sub>(378)</sub> &#x0003D; 2.31, <italic>p</italic> &#x0003D; 0.02]. The <italic>&#x003B2;</italic>-estimates indicate that on average, counterfactual explanations increased WOA by an approximated 4% compared to the control that received no explanations. Note that feature importance explanations likewise increased WOA by 2% compared to the control, but this difference was not significant for the 0.05 significance level [<italic>&#x003B2;</italic> &#x0003D; 0.02, 95%&#x02002;<italic>CI</italic>&#x02002;<italic>&#x003B2;</italic>[&#x02212;0.01, 0.05], <italic>t</italic><sub>(378)</sub> &#x0003D; 1.10, <italic>p</italic> &#x0003D; 0.27]. However, explanations had no effect on WOA when the AI recommended increasing the price estimate (<xref ref-type="fig" rid="F3">Figure 3</xref>). LMEMs are quite robust against violations of distributional assumptions (Schielzeth et al., <xref ref-type="bibr" rid="B71">2020</xref>). We nevertheless checked the residuals of WOA values for normal distribution via quantile&#x02013;quantile plots (Q&#x02013;Q plots) to determine if the residual variance was equal across conditions (homoscedasticity) and also checked the multicollinearity assumption. The normality distribution seemed to be satisfied, with some deviation from normality at the tails, which indicates that more data is located at the extremes. Levene&#x00027;s test indicated equal variances [<italic>F</italic><sub>(2)</sub> &#x0003D; 0.57, <italic>p</italic> &#x0003D; 0.56] that did not differ between the conditions, and a multicollinearity check revealed low correlation between the model terms.</p></sec>
<sec>
<title>5.3. Trust&#x02014;Trust between people and automation scale</title>
<p>To identify the underlying structure of the TPA, we performed an exploratory factor analysis (EFA) using MinRes and rotated with the Oblimin method (see footnote 1). Parallel analysis and very simple structure (VSS) indicated two factors, which is in line with previous research (Spain et al., <xref ref-type="bibr" rid="B74">2008</xref>). The first five items loaded on one factor (with 0.79 &#x02212; 0.89), and the other seven loaded on a second factor (0.56 &#x02212; 0.85), which corresponded accurately to the trust/distrust items of the scale. Internal consistency for the first five items (i.e., distrust) was excellent (&#x003B1; = 0.92, 95% CI[0.90, 0.93], &#x003C9; = 0.92, 95% CI[0.90, 0.93]) and good for the seven trust items (&#x003B1; = 0.88, 95% CI[0.86, 0.90], &#x003C9; = 0.88, 95% CI[0.86, 0.90]) according to George and Mallery (<xref ref-type="bibr" rid="B25">2019</xref>). To test <bold>H</bold><sub><bold>4</bold></sub> (feature importance leads to higher trust compared to the control), <bold>H</bold><sub><bold>5</bold></sub> (counterfactuals lead to higher trust compared to the control) and <bold>H</bold><sub><bold>6</bold></sub> (counterfactuals lead to higher trust compared to feature importance), we intended to perform two types of one-way analyses of variance (ANOVAs), once using the overall mean score, and once using mean scores for the trust and distrust factors. However, visual inspection of the distribution and a Shapiro&#x02013;Wilk test (<italic>W</italic> &#x0003D; 0.97, <italic>p</italic> &#x0003C; 0.001) revealed a non-trivial violation of the normality assumption. Thus, the ANOVA results might not have been interpretable and meaningful. Under these circumstances, a non-parametric Kruskal&#x02013;Wallis test (Kruskal and Wallis, <xref ref-type="bibr" rid="B42">1952</xref>) was carried out as it does not assume a normal distribution of the residuals. The results of the Kruskal&#x02013;Wallis test showed that the overall mean ratings for the TPA were not significantly different between the conditions [<italic>H</italic><sub>(2)</sub> &#x0003D; 1.54, <italic>p</italic> &#x0003D; 0.46]. The same was true for the factors trust [<italic>H</italic><sub>(2)</sub> &#x0003D; 5.06, <italic>p</italic> &#x0003D; 0.08] and distrust [<italic>H</italic><sub>(2)</sub> &#x0003D; 2.03, <italic>p</italic> &#x0003D; 0.36]. Since the omnibus Kruksal&#x02013;Wallis was not significant, we did not perform further <italic>post-hoc</italic> tests. <xref ref-type="fig" rid="F4">Figure 4</xref> captures the similar trust ratings for the two experimental conditions and the control.</p>
<fig id="F4" position="float">
<label>Figure 4</label>
<caption><p>Boxplots and density plots for the Trust in Automation factors trust and distrust for each condition. Black rhombuses indicate mean values. Items 1&#x02013;5 of the TPA code for distrust, items 6&#x02013;12 code for trust.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fcomp-05-1151150-g0004.tif"/>
</fig></sec></sec>
<sec sec-type="discussion" id="s6">
<title>6. Discussion</title>
<p>The experiment reported in this study demonstrates that participants generally rely on AI recommendations in low-stake decision-making tasks&#x02014;in this case, receiving AI recommendations to find an optimal price for subleasing an apartment. Regardless of the different experimental manipulations, on average, participants displayed high overall Weight of Advice scores (<italic>M</italic> &#x0003D; 0.67 &#x02212; 0.69, <italic>SD</italic> &#x0003D; 0.25 &#x02212; 0.36). A WOA of 0.70 signifies that participants adopted 70% of the AI recommendations when updating their prior beliefs to form their final estimate. This finding supports the idea that people generally rely on AI (Logg et al., <xref ref-type="bibr" rid="B51">2019</xref>).</p>
<p>The results further demonstrated that under certain conditions, explainability significantly increases AI reliance. However, in the context of our study, the effect of human-centered explanations depended on the type of decision-making the participants had to engage in. We presented participants with two kinds of recommendations: for the first type, an algorithm introduced as an AI recommended that participants <italic>increase</italic> their initially estimated apartment price. For the second type, participants were advised to <italic>decrease</italic> their initial price. The results of the experiment indicate that when the AI recommended increasing the price, human-centered explanations did not affect reliance. By contrast, in the case of recommendations to decrease the price, providing counterfactual explanations affected WOA significantly. Participants in the counterfactual condition where the AI recommended decreasing <italic>T</italic>1 relied up to 9 percentage points more on the recommendation than participants in the control that received the recommendation to increase their price. Therefore, the findings support the second hypothesis (<bold>H</bold><sub><bold>2</bold></sub>: counterfactuals will lead to higher reliance compared to the control) only for decision-making tasks where the AI recommended decreasing the price. The first hypothesis (<bold>H</bold><bold><sub>4</sub></bold><bold>H</bold><sub><bold>1</bold></sub>: feature importance will lead to higher reliance compared to the control) and the third hypothesis (<bold>H</bold><sub><bold>3</bold></sub>: counterfactuals will lead to higher reliance compared to feature importance) are not supported for either type of recommendation. We conclude that counterfactual explanations can significantly increase reliance but only under certain conditions. However, there was no significant difference between the two <italic>post-hoc</italic> explanations, although counterfactuals are arguably more human-centered since they additionally fulfill Miller (<xref ref-type="bibr" rid="B56">2019</xref>)&#x00027;s <italic>contrastivity</italic> criterion.</p>
<p>The experiment illustrated that the decision-making task with regards to increasing or decreasing a price had a significant effect on reliance. Regardless of providing explanations, participants consistently relied more on AI recommendations to decrease prices than recommendations to increase them (see <xref ref-type="fig" rid="F3">Figure 3</xref>). This seems counterintuitive at first glance since one might expect that participants would always embrace the prospect of obtaining a higher subleasing price. We argue that the two types of recommendations should be thought of as two distinct decision-making tasks and our results demonstrate how cognitive biases may affect humans in their decision-making involving AI as proposed by Kliegr et al. (<xref ref-type="bibr" rid="B38">2021</xref>). The well-studied concept of <italic>loss aversion</italic> by Tversky and Kahneman (<xref ref-type="bibr" rid="B80">1991</xref>) could account for this discrepancy and serve as an explanation attempt for our findings. Loss aversion suggests that, psychologically, people assign more utility to losses than to gains (Tversky and Kahneman, <xref ref-type="bibr" rid="B80">1991</xref>). In practical terms, this means that the dissatisfaction experienced by a person who loses $100 is greater than the satisfaction experienced by a person who gains $100. Our study design seems to satisfy the preconditions for a possible loss-aversion effect: when participants received a recommendation to increase their initial price estimate, they were likely concerned that this potential price increase would cause an unsuccessful sublease. The prospect of getting more money (gain) mattered less in this decision-making task than the possibility of not being able to sublease at all (loss). A recommendation to decrease the initial price may not have induced loss aversion in participants. When not being confronted with loss aversion, explanations seemed to convince participants that demanding less money was the right decision to successfully sublease the apartment, compared to the control where no additional explanation was present for the recommendation. This interpretation suggests that human-centered AI explanations can have an effect on reliance, but only for decision-making tasks where other contributing factors such as loss aversion are absent. Substantial work has been published about biased AI-training data but little about humans&#x00027; cognitive biases and heuristics when exposed to AI. A notable exception is the work of Lu and Yin (<xref ref-type="bibr" rid="B52">2021</xref>) that showed how people use heuristics and base their reliance on the level of agreement between the machine learning model and themselves when performance feedback was limited. Moreover, Wang et al. (<xref ref-type="bibr" rid="B83">2019</xref>) proposed a framework of how human reasoning should inform XAI to mitigate possible cognitive biases, and a recent review by Kliegr et al. (<xref ref-type="bibr" rid="B38">2021</xref>) explored to what extent biases affect human understandings of interpretable machine-learning models. We present empirical findings suggesting that the XAI community should account for possible biases and heuristics to develop genuinely human-centered explanations. Inherent biases and heuristics may be so hardwired in people that AI explanations are not convincing enough to disprove non-optimal human decision-making. If that is the case, AI may not help users to reach better decisions in circumstances where human intuition becomes too tempting for their judgment. While the interpretation of the present results under the perspective of loss aversion requires further investigation, our findings highlight the importance of biases and heuristic end-users can exhibit in actual decision-making tasks rather than proxy tasks. These biases and heuristics may result in irrational and non-optimal choices, which in turn affect the measured variables of interest, including trust and trust-related behavior (Wang et al., <xref ref-type="bibr" rid="B83">2019</xref>; Kliegr et al., <xref ref-type="bibr" rid="B38">2021</xref>). In the context of our study, the type of decision-making participants faced had a greater effect on reliance than the explanation provided by the AI. This suggests that factors other than explainability are crucial when designing human-centered AI. Cognitive biases and heuristics, such as loss aversion (Tversky and Kahneman, <xref ref-type="bibr" rid="B80">1991</xref>), framing (Tversky and Kahneman, <xref ref-type="bibr" rid="B79">1981</xref>), or confirmation bias (Wason, <xref ref-type="bibr" rid="B84">1960</xref>), could potentially undermine AI explanations.</p>
<p>Concerning trust, no significant differences were found for human-centered explanations (<xref ref-type="fig" rid="F4">Figure 4</xref>). Therefore, we reject the fourth hypothesis (<bold>H</bold><bold><sub>4</sub></bold>: feature importance will lead to higher trust compared to the control), the fifth hypothesis (<bold>H</bold><bold><sub>5</sub></bold>: counterfactuals will lead to higher trust compared to the control), and the sixth hypothesis (<bold>H</bold><bold><sub>6</sub></bold>: counterfactuals will lead to higher trust compared to feature importance). While the effect of human-centered explanations on reliance depended on the nature of the AI recommendation (increasing or decreasing the initial estimated apartment price), this dependence and the potential effect of cognitive biases and heuristics remain to be explored for trust. We operationalized trust as an attitude toward the AI system and consequently assessed users&#x00027; trust after the entire task while we observed reliance from trial-to-trial. Given the present study, our results do not indicate a consistent effect of human-centered explanations on trust. Thus, our findings are in line with other research that provided mixed evidence regarding the effect of transparency on trust (Cramer et al., <xref ref-type="bibr" rid="B13">2008</xref>; Nothdurft et al., <xref ref-type="bibr" rid="B61">2013</xref>; Cheng et al., <xref ref-type="bibr" rid="B11">2019</xref>; Ehsan et al., <xref ref-type="bibr" rid="B21">2019</xref>; Zhang et al., <xref ref-type="bibr" rid="B88">2020</xref>; Poursabzi-Sangdeh et al., <xref ref-type="bibr" rid="B65">2021</xref>). However, the conceptual distinction between trust and reliance carries significant implications for XAI evaluation and uncovers two potential challenges. First, if researchers only assess attitudinal trust via questionnaires, they could falsely assume that people will not rely on an AI system. Second, if only trust-related behavior (i.e., reliance) is measured, researchers might incorrectly deduce that people necessarily trust the system in question. Consequently, researchers and practitioners have to answer the challenging question of which of the two measures to account for and investigate when evaluating AI. Whether trust translates into reliance is more nuanced than often assumed and depends on an interaction between the operator, the automation, and the situation (K&#x000F6;rber, <xref ref-type="bibr" rid="B40">2018</xref>). It remains to be determined which factors other than trust drive AI reliance (e.g., system accuracy, perceived usefulness, cognitive biases) or whether current measurement tools originally designed to assess trust in automation also accurately capture AI trust.</p>
<p>Furthermore, we initially chose to measure overall trust, expecting a single-factor structure as proposed by Jian et al. (<xref ref-type="bibr" rid="B35">2000</xref>). However, investigating the scale&#x00027;s factor structure through exploratory factor analysis before interpreting the results implied a two-factor structure, as previously observed outside the AI context (Spain et al., <xref ref-type="bibr" rid="B74">2008</xref>). This change in the theoretical structure led us to use the TPA to measure trust and distrust as two distinct factors in the analysis. While in our study, levels of trust and distrust behaved as expected across the three conditions (i.e., high trust and low distrust), it is plausible to assume that there are situations where the difference between people&#x00027;s trust and distrust in an AI is more nuanced. Attitudes are often seen as lying along one continuum, as was initially proposed for the TPA, but past research has argued that positive and negative attitudes can co-occur (Priester and Petty, <xref ref-type="bibr" rid="B66">1996</xref>). For example, when smokers try to quit smoking, they can have a simultaneously negative and positive view toward cigarettes (Cacioppo and Berntson, <xref ref-type="bibr" rid="B10">1994</xref>). Thus, it may be necessary to distinguish between trust and distrust in studies that aim to investigate ambivalent attitudes toward AI. This distinction can be accounted for when using the two-factor structure for the TPA. On the other hand, a single-factor structure comes with the risk of oversimplifying situations and losing important nuance when using the TPA to measure trust in AI. Overall, our findings emphasize that researchers must carefully differentiate attitude from behavior and choose appropriate evaluation metrics for human-centered AI accordingly.</p></sec>
<sec id="s7">
<title>7. Limitations and future work</title>
<p>We conducted an online decision-making experiment in a domain-specific task. Future work should broaden the scope and focus on domains other than real estate to investigate if the findings of this study are transferable to different scenarios and AI systems used in practice to increase external validity. While decision-making experiments on MTurk allow high control over confounding variables and are comparable to those in laboratory settings, even in low-stakes scenarios (Amir et al., <xref ref-type="bibr" rid="B5">2012</xref>), future studies could focus on high-stakes decisions to evoke uncertainty where a more tangible loss depends on the participants&#x00027; decision to trust AI.</p>
<p>Participants were presented with AI recommendations that were expected to seem reasonably trustworthy since the recommendations were formed based on the participants&#x00027; initial estimates. However, explainability might have a greater impact on trust and reliance if the recommendations were not credible or showed greater deviations from the initial estimate. Past research has shown that people calibrate their trust based on the system&#x00027;s capabilities (Lee and See, <xref ref-type="bibr" rid="B47">2004</xref>; Zhang et al., <xref ref-type="bibr" rid="B88">2020</xref>), and often fail to rely on algorithms after learning that they are imperfect, a phenomenon called algorithm aversion (Dietvorst et al., <xref ref-type="bibr" rid="B16">2015</xref>). By providing explanations, people may better understand AI errors and factors that influence those errors (Dzindolet et al., <xref ref-type="bibr" rid="B18">2003</xref>). Future research could investigate more untrustworthy recommendations by gradually reducing the capabilities of the system (e.g., by decreasing the system&#x00027;s accuracy) and examining how explainability affects reliance and trust in cases where the AI objectively performs poorly or when there is a clear disagreement between the end-user and the AI.</p>
<p>The study design did not allow for a clear distinction between <italic>dispositional trust, situational trust</italic>, and <italic>learned trust</italic>, as suggested by Hoff and Bashir (<xref ref-type="bibr" rid="B31">2015</xref>). In addition, by measuring trust as an attitude toward the AI system after task completion, examining the effects of bias and heuristics at the level of individual trials was not possible. Trust is a dynamic process in the human&#x02013;AI interaction, and we expect that trust changes as time passes in the interaction between end-users and AI-based systems. We recommend that future studies investigate the varying manifestations of trust because they are critical for a comprehensive understanding of the human&#x02013;AI interaction. Researchers could measure AI trust before and after participants are exposed to an AI system and compare the reported trust scores (learned trust). Alternatively, they could expose participants to AI recommendations while inducing different emotional valences (situational trust). Future research could also investigate the relationship between AI trust and reliance from the perspective of the technology acceptance model (Davis, <xref ref-type="bibr" rid="B14">1989</xref>), which can be seen as a further development of Ajzen and Fishbein (<xref ref-type="bibr" rid="B4">1980</xref>)&#x00027;s work.</p></sec>
<sec sec-type="conclusions" id="s8">
<title>8. Conclusion</title>
<p>We conducted an empirical experiment demonstrating that human-centered explanations as a means for transparent AI increase reliance for specific decision-making tasks. While this provides some evidence that human-centered <italic>post-hoc</italic> explanations can be an opportunity for more transparent AI, our findings emphasize that the effect of transparency on reliance and trust is more nuanced than commonly assumed.</p>
<p>The type of decision-making task (increasing vs. decreasing a price) had a greater influence on end-users&#x00027; reliance than how the AI explained its decision did. We argue that humans may exhibit cognitive biases and apply heuristics to decision-making tasks that involve AI. So far, the discussion around bias has focused primarily on biased data and prejudice due to incorrect assumptions in the machine-learning process. The implications of potential biases and heuristics when humans are presented with explanations from AI have received little attention in the current XAI debate. Both researchers and practitioners need to be aware of such dynamics in the design for truly human-centered AI, as poor partnership between people and automation will become increasingly costly and consequential (Lee and See, <xref ref-type="bibr" rid="B47">2004</xref>).</p>
<p>In order to draw valid conclusions from experiments, XAI researchers need to be cautious when measuring the human side of the human&#x02013;AI interaction. Conceptualizing trust as an attitude and reliance as a trust-related behavior might lead to divergent results. Our study also confirmed a two-factor structure (trust and distrust) for the TPA, as previously reported outside the AI context. Given the importance of AI, as it is increasingly used to make critical decisions with far-reaching implications, meaningful evaluations in XAI research require agreed-upon metrics and appropriate measurements that have been empirically validated.</p></sec>
<sec sec-type="data-availability" id="s9">
<title>Data availability statement</title>
<p>The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found here: <ext-link ext-link-type="uri" xlink:href="https://osf.io/bs6q3/">https://osf.io/bs6q3/</ext-link>.</p></sec>
<sec sec-type="ethics-statement" id="s10">
<title>Ethics statement</title>
<p>Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.</p></sec>
<sec sec-type="author-contributions" id="s11">
<title>Author contributions</title>
<p>NS and FB contributed to the conception and design of the study and implemented the online study. NS collected the data and wrote the first draft. NS, FB, and SP performed the statistical analysis. All authors contributed to the manuscript revision, read, and approved the submitted version.</p></sec>
</body>
<back>
<sec sec-type="funding-information" id="s12">
<title>Funding</title>
<p>This research was financed entirely by our research group and the publication fund of the University of Basel for open access; we received no additional funding.</p>
</sec>
<ack><p>Special thanks to Zgjim Memeti and L&#x000E9;ane Wettstein.</p>
</ack>
<sec sec-type="COI-statement" id="conf1">
<title>Conflict of interest</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
<sec sec-type="disclaimer" id="s13">
<title>Publisher&#x00027;s note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p>
</sec>
<fn-group>
<fn id="fn0001"><p><sup>1</sup>Data, R-scripts, detailed EFA results, experimental materials (including all questions and tasks), as well as a flowchart of the inclusion and exclusion process of the experiment are available under <ext-link ext-link-type="uri" xlink:href="https://osf.io/bs6q3/">https://osf.io/bs6q3/</ext-link>.</p></fn>
</fn-group>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Abdul</surname> <given-names>A.</given-names></name> <name><surname>Vermeulen</surname> <given-names>J.</given-names></name> <name><surname>Wang</surname> <given-names>D.</given-names></name> <name><surname>Lim</surname> <given-names>B. Y.</given-names></name> <name><surname>Kankanhalli</surname> <given-names>M.</given-names></name></person-group> (<year>2018</year>). <article-title>&#x0201C;Trends and trajectories for explainable, accountable and intelligible systems: an HCI research agenda,&#x0201D;</article-title> in <source>Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI &#x00027;18</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>18</lpage>.</citation>
</ref>
<ref id="B2">
<citation citation-type="book"><person-group person-group-type="author"><collab>ACM FAccT Conference</collab></person-group> (<year>2022</year>). <article-title>&#x0201C;ACM conference on fairness, accountability, and transparency 2022 (ACM FAccT 2022) call for papers,&#x0201D;</article-title> in <source>ACM Conference on Fairness, Accountability, and Transparency 2022</source> (<publisher-loc>New York, NY</publisher-loc>).</citation>
</ref>
<ref id="B3">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Adadi</surname> <given-names>A.</given-names></name> <name><surname>Berrada</surname> <given-names>M.</given-names></name></person-group> (<year>2018</year>). <article-title>Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)</article-title>. <source>IEEE Access</source> <volume>6</volume>, <fpage>52138</fpage>&#x02013;<lpage>52160</lpage>. <pub-id pub-id-type="doi">10.1109/ACCESS.2018.2870052</pub-id></citation>
</ref>
<ref id="B4">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ajzen</surname> <given-names>I.</given-names></name> <name><surname>Fishbein</surname> <given-names>M.</given-names></name></person-group> (<year>1980</year>). <source>Understanding Attitudes and Predicting Social Behavior</source>. <publisher-loc>Englewood Cliffs, NJ</publisher-loc>: <publisher-name>Prentice-Hall</publisher-name>.</citation>
</ref>
<ref id="B5">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Amir</surname> <given-names>O.</given-names></name> <name><surname>Rand</surname> <given-names>D. G.</given-names></name> <name><surname>Gal</surname> <given-names>Y. K.</given-names></name></person-group> (<year>2012</year>). <article-title>Economic games on the internet: the effect of $1 stakes</article-title>. <source>PLoS ONE</source> <volume>7</volume>, <fpage>e31461</fpage>. <pub-id pub-id-type="doi">10.1371/journal.pone.0031461</pub-id><pub-id pub-id-type="pmid">22363651</pub-id></citation></ref>
<ref id="B6">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Arrieta</surname> <given-names>A. B.</given-names></name> <name><surname>Diaz-Rodriguez</surname> <given-names>N.</given-names></name> <name><surname>Ser</surname> <given-names>J. D.</given-names></name> <name><surname>Bennetot</surname> <given-names>A.</given-names></name> <name><surname>Tabik</surname> <given-names>S.</given-names></name> <name><surname>Barbado</surname> <given-names>A.</given-names></name> <etal/></person-group>. (<year>2020</year>). <article-title>Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI</article-title>. <source>Inform. Fusion</source> <volume>58</volume>, <fpage>82</fpage>&#x02013;<lpage>115</lpage>. <pub-id pub-id-type="doi">10.1016/j.inffus.2019.12.012</pub-id></citation>
</ref>
<ref id="B7">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bates</surname> <given-names>D.</given-names></name> <name><surname>Mochler</surname> <given-names>M.</given-names></name> <name><surname>Bolker</surname> <given-names>B.</given-names></name> <name><surname>Walker</surname> <given-names>S.</given-names></name></person-group> (<year>2015</year>). <article-title>Fitting linear mixed-effects models using lme4</article-title>. <source>J. Stat. Softw.</source> <volume>67</volume>, <fpage>1</fpage>&#x02013;<lpage>48</lpage>. <pub-id pub-id-type="doi">10.18637/jss.v067.i01</pub-id></citation>
</ref>
<ref id="B8">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Biran</surname> <given-names>O.</given-names></name> <name><surname>Cotton</surname> <given-names>C.</given-names></name></person-group> (<year>2017</year>). <article-title>&#x0201C;Explanation and justification in machine learning: a survey,&#x0201D;</article-title> in <source>IJCAI-17 Workshop on Explainable AI (XAI)</source> (<publisher-loc>Melbourne</publisher-loc>), <fpage>8</fpage>&#x02013;<lpage>13</lpage>.</citation>
</ref>
<ref id="B9">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Bu&#x000E7;inca</surname> <given-names>Z.</given-names></name> <name><surname>Lin</surname> <given-names>P.</given-names></name> <name><surname>Gajos</surname> <given-names>K. Z.</given-names></name> <name><surname>Glassman</surname> <given-names>E. L.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems,&#x0201D;</article-title> in <source>Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI &#x00027;20</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>454</fpage>&#x02013;<lpage>464</lpage>.</citation>
</ref>
<ref id="B10">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cacioppo</surname> <given-names>J. T.</given-names></name> <name><surname>Berntson</surname> <given-names>G. G.</given-names></name></person-group> (<year>1994</year>). <article-title>Relationship between attitudes and evaluative space: a critical review, with emphasis on the separability of positive and negative substrates</article-title>. <source>Psychol. Bull.</source> <volume>115</volume>, <fpage>401</fpage>&#x02013;<lpage>423</lpage>.</citation>
</ref>
<ref id="B11">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Cheng</surname> <given-names>H.-F.</given-names></name> <name><surname>Wang</surname> <given-names>R.</given-names></name> <name><surname>Zhang</surname> <given-names>Z.</given-names></name> <name><surname>O&#x00027;Connell</surname> <given-names>F.</given-names></name> <name><surname>Gray</surname> <given-names>T.</given-names></name> <name><surname>Harper</surname> <given-names>F. M.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>&#x0201C;Explaining decision-making algorithms through UI: strategies to help non-expert stakeholders,&#x0201D;</article-title> in <source>Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI &#x00027;19</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>12</lpage>.</citation>
</ref>
<ref id="B12">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Chopra</surname> <given-names>K.</given-names></name> <name><surname>Wallace</surname> <given-names>W. A.</given-names></name></person-group> (<year>2003</year>). <article-title>&#x0201C;Trust in electronic environments,&#x0201D;</article-title> in <source>Proceedings of the 36th Annual Hawaii International Conference on System Sciences, HICSS &#x00027;03</source> (<publisher-loc>Los Alamitos, CA</publisher-loc>: <publisher-name>IEEE Computer Society Press, IEEE Computer Society Press</publisher-name>), <fpage>10</fpage>&#x02013;<lpage>15</lpage>.</citation>
</ref>
<ref id="B13">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cramer</surname> <given-names>H.</given-names></name> <name><surname>Evers</surname> <given-names>V.</given-names></name> <name><surname>Ramlal</surname> <given-names>S.</given-names></name> <name><surname>Someren</surname> <given-names>M.</given-names></name> <name><surname>Rutledge</surname> <given-names>L.</given-names></name> <name><surname>Stash</surname> <given-names>N.</given-names></name> <etal/></person-group>. (<year>2008</year>). <article-title>The effects of transparency on trust in and acceptance of a content-based art recommender</article-title>. <source>User Model. User Adapt. Interact.</source> <volume>18</volume>, <fpage>455</fpage>&#x02013;<lpage>496</lpage>. <pub-id pub-id-type="doi">10.1007/s11257-008-9051-3</pub-id><pub-id pub-id-type="pmid">33852020</pub-id></citation></ref>
<ref id="B14">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Davis</surname> <given-names>F. D.</given-names></name></person-group> (<year>1989</year>). <article-title>Perceived usefulness, perceived ease of use, and user acceptance of information technology</article-title>. <source>MIS Q.</source> <volume>13</volume>, <fpage>319</fpage>&#x02013;<lpage>340</lpage>.</citation>
</ref>
<ref id="B15">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>De Graaf</surname> <given-names>M. M. A.</given-names></name> <name><surname>Malle</surname> <given-names>B. F.</given-names></name></person-group> (<year>2018</year>). <article-title>&#x0201C;People&#x00027;s judgments of human and robot behaviors: a robust set of behaviors and some discrepancies,&#x0201D;</article-title> in <source>Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, HRI &#x00027;18</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>97</fpage>&#x02013;<lpage>98</lpage>.</citation>
</ref>
<ref id="B16">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Dietvorst</surname> <given-names>B. J.</given-names></name> <name><surname>Simmons</surname> <given-names>J. P.</given-names></name> <name><surname>Massey</surname> <given-names>C.</given-names></name></person-group> (<year>2015</year>). <article-title>Algorithm aversion: people erroneously avoid algorithms after seeing them err</article-title>. <source>J. Exp. Psychol. Gen.</source> <volume>144</volume>, <fpage>114</fpage>&#x02013;<lpage>126</lpage>. <pub-id pub-id-type="doi">10.1037/xge0000033</pub-id><pub-id pub-id-type="pmid">25401381</pub-id></citation></ref>
<ref id="B17">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Du</surname> <given-names>M.</given-names></name> <name><surname>Liu</surname> <given-names>N.</given-names></name> <name><surname>Hu</surname> <given-names>X.</given-names></name></person-group> (<year>2019</year>). <article-title>Techniques for interpretable machine learning</article-title>. <source>Commun. ACM</source> <volume>63</volume>, <fpage>68</fpage>&#x02013;<lpage>77</lpage>. <pub-id pub-id-type="doi">10.1145/3359786</pub-id></citation>
</ref>
<ref id="B18">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Dzindolet</surname> <given-names>M. T.</given-names></name> <name><surname>Peterson</surname> <given-names>S. A.</given-names></name> <name><surname>Pomranky</surname> <given-names>R. A.</given-names></name> <name><surname>Pierce</surname> <given-names>L. G.</given-names></name> <name><surname>Beck</surname> <given-names>H. P.</given-names></name></person-group> (<year>2003</year>). <article-title>The role of trust in automation reliance</article-title>. <source>Int. J. Hum. Comput. Stud.</source> <volume>58</volume>, <fpage>697</fpage>&#x02013;<lpage>718</lpage>. <pub-id pub-id-type="doi">10.1016/S1071-5819(03)00038-7</pub-id></citation>
</ref>
<ref id="B19">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ehsan</surname> <given-names>U.</given-names></name> <name><surname>Liao</surname> <given-names>Q. V.</given-names></name> <name><surname>Muller</surname> <given-names>M.</given-names></name> <name><surname>Riedl</surname> <given-names>M. O.</given-names></name> <name><surname>Weisz</surname> <given-names>J. D.</given-names></name></person-group> (<year>2021a</year>). <article-title>&#x0201C;Expanding explainability: towards social transparency in AI systems,&#x0201D;</article-title> in <source>Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>19</lpage>.</citation>
</ref>
<ref id="B20">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ehsan</surname> <given-names>U.</given-names></name> <name><surname>Riedl</surname> <given-names>M. O.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Human-centered explainable AI: towards a reflective sociotechnical approach,&#x0201D;</article-title> in <source>HCI International 2020-Late Breaking Papers: Multimodality and Intelligence: 22nd HCI International Conference, HCII 2020</source> (<publisher-loc>Copenhagen</publisher-loc>), <fpage>449</fpage>&#x02013;<lpage>466</lpage>.</citation>
</ref>
<ref id="B21">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ehsan</surname> <given-names>U.</given-names></name> <name><surname>Tambwekar</surname> <given-names>P.</given-names></name> <name><surname>Chan</surname> <given-names>L.</given-names></name> <name><surname>Harrison</surname> <given-names>B.</given-names></name> <name><surname>Riedl</surname> <given-names>M. O.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;Automated rationale generation: a technique for explainable ai and its effects on human perceptions,&#x0201D;</article-title> in <source>Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI &#x00027;19</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>263</fpage>&#x02013;<lpage>274</lpage>.</citation>
</ref>
<ref id="B22">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Ehsan</surname> <given-names>U.</given-names></name> <name><surname>Wintersberger</surname> <given-names>P.</given-names></name> <name><surname>Liao</surname> <given-names>Q. V.</given-names></name> <name><surname>Mara</surname> <given-names>M.</given-names></name> <name><surname>Streit</surname> <given-names>M.</given-names></name> <name><surname>Wachter</surname> <given-names>S.</given-names></name> <etal/></person-group>. (<year>2021b</year>). <article-title>&#x0201C;Operationalizing human-centered perspectives in explainable AI,&#x0201D;</article-title> in <source>Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>Association for Computing Machinery</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>6</lpage>.</citation>
</ref>
<ref id="B23">
<citation citation-type="journal"><person-group person-group-type="author"><collab>European Parliament and Council of the European Union</collab></person-group> (<year>2016</year>). <source>Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons With Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation)</source>.</citation>
</ref>
<ref id="B24">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ferreira</surname> <given-names>J. J.</given-names></name> <name><surname>Monteiro</surname> <given-names>M. S.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;What are people doing about XAI user experience? A survey on ai explainability research and practice,&#x0201D;</article-title> in <source>Design, User Experience, and Usability. Design for Contemporary Interactive Environments</source>, eds A. Marcus and E. Rosenzweig (Cham: Springer International Publishing), <fpage>56</fpage>&#x02013;<lpage>73</lpage>.</citation>
</ref>
<ref id="B25">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>George</surname> <given-names>D.</given-names></name> <name><surname>Mallery</surname> <given-names>P.</given-names></name></person-group> (<year>2019</year>). <source>IBM SPSS Statistics 26 Step by Step: A Simple Guide and Reference, 16th Edn</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>Routledge</publisher-name>.</citation>
</ref>
<ref id="B26">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Gino</surname> <given-names>F.</given-names></name> <name><surname>Moore</surname> <given-names>D. A.</given-names></name></person-group> (<year>2007</year>). <article-title>Effects of task difficulty on use of advice</article-title>. <source>J. Behav. Decis. Mak.</source> <volume>20</volume>, <fpage>21</fpage>&#x02013;<lpage>35</lpage>. <pub-id pub-id-type="doi">10.1002/bdm.539</pub-id></citation>
</ref>
<ref id="B27">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Gutzwiller</surname> <given-names>R. S.</given-names></name> <name><surname>Chiou</surname> <given-names>E. K.</given-names></name> <name><surname>Craig</surname> <given-names>S. D.</given-names></name> <name><surname>Lewis</surname> <given-names>C. M.</given-names></name> <name><surname>Lematta</surname> <given-names>G. J.</given-names></name> <name><surname>Hsiung</surname> <given-names>C.-P.</given-names></name></person-group> (<year>2019</year>). &#x0201C;Positive bias in the &#x0201C;trust in automated systems survey&#x0201D;? An examination of the Jian et al. (2000) scale,&#x0201D; in <source>Proceedings of the Human Factors and Ergonomics Society Annual Meeting</source>, <fpage>217</fpage>&#x02013;<lpage>221</lpage>.</citation>
</ref>
<ref id="B28">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Hartmann</surname> <given-names>M.</given-names></name></person-group> (<year>2020</year>). <source>Vertrauen - Die unsichtbare Macht</source>. <publisher-loc>Berlin</publisher-loc>: <publisher-name>Fischer Verlag</publisher-name>.</citation>
</ref>
<ref id="B29">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Harvey</surname> <given-names>N.</given-names></name> <name><surname>Fischer</surname> <given-names>I.</given-names></name></person-group> (<year>1997</year>). <article-title>Taking advice: accepting help, improving judgment, and sharing responsibility</article-title>. <source>Organ. Behav. Hum. Decis. Process.</source> <volume>70</volume>, <fpage>117</fpage>&#x02013;<lpage>133</lpage>.</citation>
</ref>
<ref id="B30">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hilton</surname> <given-names>D. J.</given-names></name></person-group> (<year>1990</year>). <article-title>Conversational processes and causal explanation</article-title>. <source>Psychol. Bull.</source> <volume>107</volume>, <fpage>65</fpage>&#x02013;<lpage>81</lpage>.</citation>
</ref>
<ref id="B31">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hoff</surname> <given-names>K. A.</given-names></name> <name><surname>Bashir</surname> <given-names>M.</given-names></name></person-group> (<year>2015</year>). <article-title>Trust in automation: integrating empirical evidence on factors that influence trust</article-title>. <source>Hum. Fact.</source> <volume>57</volume>, <fpage>407</fpage>&#x02013;<lpage>434</lpage>. <pub-id pub-id-type="doi">10.1177/0018720814547570</pub-id><pub-id pub-id-type="pmid">25875432</pub-id></citation></ref>
<ref id="B32">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hoffman</surname> <given-names>R. R.</given-names></name> <name><surname>Mueller</surname> <given-names>S. T.</given-names></name> <name><surname>Klein</surname> <given-names>G.</given-names></name> <name><surname>Litman</surname> <given-names>J.</given-names></name></person-group> (<year>2019</year>). <article-title>Metrics for explainable AI: challenges and prospects</article-title>. <source>arXiv preprint arxiv: 1812.04608</source>. <pub-id pub-id-type="doi">10.48550/arXiv.1812.04608</pub-id></citation>
</ref>
<ref id="B33">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Hong</surname> <given-names>S. R.</given-names></name> <name><surname>Hullman</surname> <given-names>J.</given-names></name> <name><surname>Bertini</surname> <given-names>E.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Human factors in model interpretability: industry practices, challenges, and needs,&#x0201D;</article-title> in <source>Proceedings of the ACM on Human-Computer Interaction</source> (<publisher-loc>New York, NY</publisher-loc>).</citation>
</ref>
<ref id="B34">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Jacovi</surname> <given-names>A.</given-names></name> <name><surname>Marasovi&#x00107;</surname> <given-names>A.</given-names></name> <name><surname>Miller</surname> <given-names>T.</given-names></name> <name><surname>Goldberg</surname> <given-names>Y.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI,&#x0201D;</article-title> in <source>Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT &#x00027;21</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>624</fpage>&#x02013;<lpage>635</lpage>.</citation>
</ref>
<ref id="B35">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jian</surname> <given-names>J.-Y.</given-names></name> <name><surname>Bisantz</surname> <given-names>A. M.</given-names></name> <name><surname>Drury</surname> <given-names>C. G.</given-names></name></person-group> (<year>2000</year>). <article-title>Foundations for an empirically determined scale of trust in automated systems</article-title>. <source>Int. J. Cogn. Ergon.</source> <volume>4</volume>, <fpage>53</fpage>&#x02013;<lpage>71</lpage>. <pub-id pub-id-type="doi">10.1207/S15327566IJCE0401_04</pub-id><pub-id pub-id-type="pmid">27885969</pub-id></citation></ref>
<ref id="B36">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kirlik</surname> <given-names>A.</given-names></name></person-group> (<year>1993</year>). <article-title>Modeling strategic behavior in human-automation interaction: why an &#x0201C;aid&#x0201D; can (and should) go unused</article-title>. <source>Hum. Fact.</source> <volume>35</volume>, <fpage>221</fpage>&#x02013;<lpage>242</lpage>.<pub-id pub-id-type="pmid">8349287</pub-id></citation></ref>
<ref id="B37">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Kizilcec</surname> <given-names>R. F.</given-names></name></person-group> (<year>2016</year>). <article-title>&#x0201C;How much information? Effects of transparency on trust in an algorithmic interface,&#x0201D;</article-title> in <source>Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>2390</fpage>&#x02013;<lpage>2395</lpage>.</citation>
</ref>
<ref id="B38">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kliegr</surname> <given-names>T.</given-names></name> <name><surname>Stepan Bahnik Furnkranz</surname> <given-names>J.</given-names></name></person-group> (<year>2021</year>). <article-title>A review of possible effects of cognitive biases on interpretation of rule-based machine learning models</article-title>. <source>Artif. Intell.</source> <volume>295</volume>, <fpage>103458</fpage>. <pub-id pub-id-type="doi">10.1016/j.artint.2021.103458</pub-id></citation>
</ref>
<ref id="B39">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Kocielnik</surname> <given-names>R.</given-names></name> <name><surname>Amershi</surname> <given-names>S.</given-names></name> <name><surname>Bennett</surname> <given-names>P. N.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems,&#x0201D;</article-title> in <source>Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>14</lpage>.</citation>
</ref>
<ref id="B40">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>K&#x000F6;rber</surname> <given-names>M.</given-names></name></person-group> (<year>2018</year>). <article-title>&#x0201C;Theoretical considerations and development of a questionnaire to measure trust in automation,&#x0201D;</article-title> in <source>Proceedings of the 20th Congress of the International Ergonomics Association, IEA &#x00027;18</source> (<publisher-loc>Cham</publisher-loc>: <publisher-name>Springer International Publishing</publisher-name>), <fpage>13</fpage>&#x02013;<lpage>30</lpage>.<pub-id pub-id-type="pmid">26384314</pub-id></citation></ref>
<ref id="B41">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Krause</surname> <given-names>J.</given-names></name> <name><surname>Perer</surname> <given-names>A.</given-names></name> <name><surname>Ng</surname> <given-names>K.</given-names></name></person-group> (<year>2016</year>). <article-title>&#x0201C;Interacting with predictions: visual inspection of black-box machine learning models,&#x0201D;</article-title> in <source>Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>5686</fpage>&#x02013;<lpage>5697</lpage>.<pub-id pub-id-type="pmid">31514137</pub-id></citation></ref>
<ref id="B42">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kruskal</surname> <given-names>W. H.</given-names></name> <name><surname>Wallis</surname> <given-names>W. A.</given-names></name></person-group> (<year>1952</year>). <article-title>Use of ranks in one-criterion variance analysis</article-title>. <source>J. Am. Stat. Assoc.</source> <volume>47</volume>, <fpage>583</fpage>&#x02013;<lpage>621</lpage>.</citation>
</ref>
<ref id="B43">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Kulesza</surname> <given-names>T.</given-names></name> <name><surname>Burnett</surname> <given-names>M.</given-names></name> <name><surname>Wong</surname> <given-names>W.-K.</given-names></name> <name><surname>Stumpf</surname> <given-names>S.</given-names></name></person-group> (<year>2015</year>). <article-title>&#x0201C;Principles of explanatory debugging to personalize interactive machine learning,&#x0201D;</article-title> in <source>Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI &#x00027;15</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>126</fpage>&#x02013;<lpage>137</lpage>.</citation>
</ref>
<ref id="B44">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Lai</surname> <given-names>V.</given-names></name> <name><surname>Tan</surname> <given-names>C.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;On human predictions with explanations and predictions of machine learning models: a case study on deception detection,&#x0201D;</article-title> in <source>Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* &#x00027;19</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>29</fpage>&#x02013;<lpage>38</lpage>.</citation>
</ref>
<ref id="B45">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Langer</surname> <given-names>M.</given-names></name> <name><surname>Oster</surname> <given-names>D.</given-names></name> <name><surname>Speith</surname> <given-names>T.</given-names></name> <name><surname>K&#x000E4;stner</surname> <given-names>L.</given-names></name> <name><surname>Hermanns</surname> <given-names>H.</given-names></name> <name><surname>Schmidt</surname> <given-names>E.</given-names></name> <etal/></person-group>. (<year>2021</year>). <article-title>What do we want from explainable artificial intelligence (XAI)? A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research</article-title>. <source>Artif. Intell.</source> <volume>296</volume>, <fpage>103473</fpage>. <pub-id pub-id-type="doi">10.1016/j.artint.2021.103473</pub-id></citation>
</ref>
<ref id="B46">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>LeCun</surname> <given-names>Y.</given-names></name> <name><surname>Bengio</surname> <given-names>Y.</given-names></name> <name><surname>Hinton</surname> <given-names>G.</given-names></name></person-group> (<year>2015</year>). <article-title>Deep learning</article-title>. <source>Nature</source> <volume>521</volume>, <fpage>436</fpage>&#x02013;<lpage>444</lpage>. <pub-id pub-id-type="doi">10.1038/nature14539</pub-id><pub-id pub-id-type="pmid">26017442</pub-id></citation></ref>
<ref id="B47">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lee</surname> <given-names>J. D.</given-names></name> <name><surname>See</surname> <given-names>K. A.</given-names></name></person-group> (<year>2004</year>). <article-title>Trust in automation: designing for appropriate reliance</article-title>. <source>Hum. Fact.</source> <volume>46</volume>, <fpage>50</fpage>&#x02013;<lpage>80</lpage>. <pub-id pub-id-type="doi">10.1518/hfes.46.1.50.30392</pub-id><pub-id pub-id-type="pmid">15151155</pub-id></citation></ref>
<ref id="B48">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Liao</surname> <given-names>Q. V.</given-names></name> <name><surname>Gruen</surname> <given-names>D.</given-names></name> <name><surname>Miller</surname> <given-names>S.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Questioning the AI: informing design practices for explainable ai user experiences,&#x0201D;</article-title> in <source>Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI &#x00027;20</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>15</lpage>.</citation>
</ref>
<ref id="B49">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lipton</surname> <given-names>Z. C.</given-names></name></person-group> (<year>2018a</year>). <article-title>The mythos of model interpretability</article-title>. <source>Commun. ACM</source> <volume>61</volume>, <fpage>36</fpage>&#x02013;<lpage>43</lpage>. <pub-id pub-id-type="doi">10.1145/3233231</pub-id></citation>
</ref>
<ref id="B50">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lipton</surname> <given-names>Z. C.</given-names></name></person-group> (<year>2018b</year>). <article-title>The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery</article-title>. <source>Queue</source> <volume>16</volume>, <fpage>31</fpage>&#x02013;<lpage>57</lpage>. <pub-id pub-id-type="doi">10.1145/3236386.3241340</pub-id></citation>
</ref>
<ref id="B51">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Logg</surname> <given-names>J. M.</given-names></name> <name><surname>Minson</surname> <given-names>J. A.</given-names></name> <name><surname>Moore</surname> <given-names>D. A.</given-names></name></person-group> (<year>2019</year>). <article-title>Algorithm appreciation: people prefer algorithmic to human judgment</article-title>. <source>Organ. Behav. Hum. Decis. Process.</source> <volume>151</volume>, <fpage>90</fpage>&#x02013;<lpage>103</lpage>. <pub-id pub-id-type="doi">10.1016/j.obhdp.2018.12.005</pub-id></citation>
</ref>
<ref id="B52">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Lu</surname> <given-names>Z.</given-names></name> <name><surname>Yin</surname> <given-names>M.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Human reliance on machine learning models when performance feedback is limited: heuristics and risks,&#x0201D;</article-title> in <source>Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>16</lpage>.</citation>
</ref>
<ref id="B53">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Mayer</surname> <given-names>R. C.</given-names></name> <name><surname>Davis</surname> <given-names>J. H.</given-names></name> <name><surname>Schoorman</surname> <given-names>F. D.</given-names></name></person-group> (<year>1995</year>). <article-title>An integrative model of organizational trust</article-title>. <source>Acad. Manage. Rev.</source> <volume>20</volume>, <fpage>709</fpage>&#x02013;<lpage>734</lpage>.</citation>
</ref>
<ref id="B54">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Merritt</surname> <given-names>S. M.</given-names></name></person-group> (<year>2011</year>). <article-title>Affective processes in human&#x02013;automation interactions</article-title>. <source>Hum. Fact.</source> <volume>53</volume>, <fpage>356</fpage>&#x02013;<lpage>370</lpage>. <pub-id pub-id-type="doi">10.1177/0018720811411912</pub-id><pub-id pub-id-type="pmid">21901933</pub-id></citation></ref>
<ref id="B55">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Miller</surname> <given-names>D.</given-names></name> <name><surname>Johns</surname> <given-names>M.</given-names></name> <name><surname>Mok</surname> <given-names>B.</given-names></name> <name><surname>Gowda</surname> <given-names>N.</given-names></name> <name><surname>Sirkin</surname> <given-names>D.</given-names></name> <name><surname>Lee</surname> <given-names>K.</given-names></name> <etal/></person-group>. (<year>2016</year>). <article-title>&#x0201C;Behavioral measurement of trust in automation: the trust fall,&#x0201D;</article-title> in <source>Proceedings of the Human Factors and Ergonomics Society Annual Meeting</source> (<publisher-loc>Washington, DC</publisher-loc>), <fpage>1849</fpage>&#x02013;<lpage>1853</lpage>.</citation>
</ref>
<ref id="B56">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Miller</surname> <given-names>T.</given-names></name></person-group> (<year>2019</year>). <article-title>Explanation in artificial intelligence: insights from the social sciences</article-title>. <source>Artif. Intell.</source> <volume>267</volume>, <fpage>1</fpage>&#x02013;<lpage>38</lpage>. <pub-id pub-id-type="doi">10.1016/j.artint.2018.07.007</pub-id></citation>
</ref>
<ref id="B57">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Mittelstadt</surname> <given-names>B.</given-names></name> <name><surname>Russell</surname> <given-names>C.</given-names></name> <name><surname>Wachter</surname> <given-names>S.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;Explaining explanations in AI,&#x0201D;</article-title> in <source>Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* &#x00027;19</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>279</fpage>&#x02013;<lpage>288</lpage>.</citation>
</ref>
<ref id="B58">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Mohseni</surname> <given-names>S.</given-names></name> <name><surname>Zarei</surname> <given-names>N.</given-names></name> <name><surname>Ragan</surname> <given-names>E. D.</given-names></name></person-group> (<year>2020</year>). <article-title>A multidisciplinary survey and framework for design and evaluation of explainable AI systems</article-title>. <source>arXiv preprint arxiv:1811.11839</source>.</citation>
</ref>
<ref id="B59">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Mothilal</surname> <given-names>R. K.</given-names></name> <name><surname>Sharma</surname> <given-names>A.</given-names></name> <name><surname>Tan</surname> <given-names>C.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Explaining machine learning classifiers through diverse counterfactual explanations,&#x0201D;</article-title> in <source>Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* &#x00027;20</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>607</fpage>&#x02013;<lpage>617</lpage>.</citation>
</ref>
<ref id="B60">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Mucha</surname> <given-names>H.</given-names></name> <name><surname>Robert</surname> <given-names>S.</given-names></name> <name><surname>Breitschwerdt</surname> <given-names>R.</given-names></name> <name><surname>Fellmann</surname> <given-names>M.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Interfaces for explanations in human-AI interaction: proposing a design evaluation approach,&#x0201D;</article-title> in <source>Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), 1 &#x02013;6.</citation>
</ref>
<ref id="B61">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Nothdurft</surname> <given-names>F.</given-names></name> <name><surname>Heinroth</surname> <given-names>T.</given-names></name> <name><surname>Minker</surname> <given-names>W.</given-names></name></person-group> (<year>2013</year>). <article-title>&#x0201C;The impact of explanation dialogues on human-computer trust,&#x0201D;</article-title> in <source>Proceedings, Part III, of the 15th International Conference on Human-Computer Interaction. Users and Contexts of Use</source> (<publisher-loc>Berlin; Heidelberg</publisher-loc>: <publisher-name>Springer-Verlag</publisher-name>), <fpage>59</fpage>&#x02013;<lpage>67</lpage>.</citation>
</ref>
<ref id="B62">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Papenmeier</surname> <given-names>A.</given-names></name> <name><surname>Kern</surname> <given-names>D.</given-names></name> <name><surname>Englebienne</surname> <given-names>G.</given-names></name> <name><surname>Seifert</surname> <given-names>C.</given-names></name></person-group> (<year>2022</year>). <article-title>It&#x00027;s complicated: the relationship between user trust, model accuracy and explanations in AI</article-title>. <source>ACM Trans. Comput. Hum. Interact.</source> <volume>29</volume>, <fpage>1</fpage>&#x02013;<lpage>33</lpage>. <pub-id pub-id-type="doi">10.1145/3495013</pub-id></citation>
</ref>
<ref id="B63">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Parasuraman</surname> <given-names>R.</given-names></name> <name><surname>Manzey</surname> <given-names>D. H.</given-names></name></person-group> (<year>2010</year>). <article-title>Complacency and bias in human use of automation: an attentional integration</article-title>. <source>Hum. Fact.</source> <volume>52</volume>, <fpage>381</fpage>&#x02013;<lpage>410</lpage>. <pub-id pub-id-type="doi">10.1177/0018720810376055</pub-id><pub-id pub-id-type="pmid">21077562</pub-id></citation></ref>
<ref id="B64">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Parasuraman</surname> <given-names>R.</given-names></name> <name><surname>Riley</surname> <given-names>V.</given-names></name></person-group> (<year>1997</year>). <article-title>Humans and automation: use, misuse, disuse, abuse</article-title>. <source>Hum. Fact.</source> <volume>39</volume>, <fpage>230</fpage>&#x02013;<lpage>253</lpage>.<pub-id pub-id-type="pmid">18689046</pub-id></citation></ref>
<ref id="B65">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Poursabzi-Sangdeh</surname> <given-names>F.</given-names></name> <name><surname>Goldstein</surname> <given-names>D. G.</given-names></name> <name><surname>Hofman</surname> <given-names>J. M.</given-names></name> <name><surname>Wortman Vaughan</surname> <given-names>J. W.</given-names></name> <name><surname>Wallach</surname> <given-names>H.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Manipulating and measuring model interpretability,&#x0201D;</article-title> in <source>Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI &#x00027;21</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>52</lpage>.</citation>
</ref>
<ref id="B66">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Priester</surname> <given-names>J. R.</given-names></name> <name><surname>Petty</surname> <given-names>R. E.</given-names></name></person-group> (<year>1996</year>). <article-title>The gradual threshold model of ambivalence: relating the positive and negative bases of attitudes to subjective ambivalence</article-title>. <source>J. Pers. Soc. Psychol.</source> <volume>71</volume>, <fpage>431</fpage>.<pub-id pub-id-type="pmid">8831157</pub-id></citation></ref>
<ref id="B67">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rousseau</surname> <given-names>D. M.</given-names></name> <name><surname>Sitkin</surname> <given-names>S. B.</given-names></name> <name><surname>Burt</surname> <given-names>R. S.</given-names></name> <name><surname>Camerer</surname> <given-names>C.</given-names></name></person-group> (<year>1998</year>). <article-title>Not so different after all: a cross-discipline view of trust</article-title>. <source>Acad. Manage. Rev.</source> <volume>23</volume>, <fpage>393</fpage>&#x02013;<lpage>404</lpage>.</citation>
</ref>
<ref id="B68">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Samek</surname> <given-names>W.</given-names></name> <name><surname>Montavon</surname> <given-names>G.</given-names></name> <name><surname>Vedaldi</surname> <given-names>A.</given-names></name> <name><surname>Hansen</surname> <given-names>L. K.</given-names></name> <name><surname>M&#x000FC;ller</surname> <given-names>K.-R.</given-names></name></person-group> (<year>2019</year>). <source>Explainable AI: Interpreting, Explaining and Visualizing Deep Learning</source>. <publisher-loc>Berlin</publisher-loc>: <publisher-name>Springer</publisher-name>.</citation>
</ref>
<ref id="B69">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sanneman</surname> <given-names>L.</given-names></name> <name><surname>Shah</surname> <given-names>J. A.</given-names></name></person-group> (<year>2022</year>). <article-title>The situation awareness framework for explainable AI (safe-AI) and human factors considerations for XAI systems</article-title>. <source>Int. J. Hum. Comput. Interact.</source> <volume>38</volume>, <fpage>1772</fpage>&#x02013;<lpage>1788</lpage>. <pub-id pub-id-type="doi">10.1080/10447318.2022.2081282</pub-id></citation>
</ref>
<ref id="B70">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Scharowski</surname> <given-names>N.</given-names></name> <name><surname>Perrig</surname> <given-names>S. A.</given-names></name> <name><surname>von Felten</surname> <given-names>N.</given-names></name> <name><surname>Br&#x000FC;hlmann</surname> <given-names>F.</given-names></name></person-group> (<year>2022</year>). <article-title>&#x0201C;Trust and reliance in XAI-distinguishing between attitudinal and behavioral measures,&#x0201D;</article-title> in <source>CHI TRAIT Workshop</source> (<publisher-loc>New York, NY</publisher-loc>).</citation>
</ref>
<ref id="B71">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Schielzeth</surname> <given-names>H.</given-names></name> <name><surname>Dingemanse</surname> <given-names>N. J.</given-names></name> <name><surname>Nakagawa</surname> <given-names>S.</given-names></name> <name><surname>Westneat</surname> <given-names>D. F.</given-names></name> <name><surname>Allegue</surname> <given-names>H.</given-names></name> <name><surname>Teplitsky</surname> <given-names>C.</given-names></name> <etal/></person-group>. (<year>2020</year>). <article-title>Robustness of linear mixed-effects models to violations of distributional assumptions</article-title>. <source>Methods Ecol. Evol.</source> <volume>11</volume>, <fpage>1141</fpage>&#x02013;<lpage>1152</lpage>. <pub-id pub-id-type="doi">10.1111/2041-210X.13434</pub-id></citation>
</ref>
<ref id="B72">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Shmueli</surname> <given-names>G.</given-names></name></person-group> (<year>2010</year>). <article-title>To explain or to predict?</article-title> <source>Stat. Sci.</source> <volume>25</volume>, <fpage>289</fpage>&#x02013;<lpage>310</lpage>. <pub-id pub-id-type="doi">10.1214/10-STS330</pub-id></citation>
</ref>
<ref id="B73">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Solso</surname> <given-names>R. L.</given-names></name> <name><surname>MacLin</surname> <given-names>M. K.</given-names></name> <name><surname>MacLin</surname> <given-names>O. H.</given-names></name></person-group> (<year>2005</year>). <source>Cognitive Psychology</source>. <publisher-loc>Auckland</publisher-loc>: <publisher-name>Pearson Education</publisher-name>.</citation>
</ref>
<ref id="B74">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Spain</surname> <given-names>R. D.</given-names></name> <name><surname>Bustamante</surname> <given-names>E. A.</given-names></name> <name><surname>Bliss</surname> <given-names>J. P.</given-names></name></person-group> (<year>2008</year>). <article-title>&#x0201C;Towards an empirically developed scale for system trust: take two,&#x0201D;</article-title> in <source>Proceedings of the Human Factors and Ergonomics Society Annual Meeting</source> (<publisher-loc>Los Angeles, CA</publisher-loc>: <publisher-name>Sage Publications</publisher-name>), <fpage>1335</fpage>&#x02013;<lpage>1339</lpage>.</citation>
</ref>
<ref id="B75">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Stephanidis</surname> <given-names>C.</given-names></name> <name><surname>Salvendy</surname> <given-names>G.</given-names></name> <name><surname>Antona</surname> <given-names>M.</given-names></name> <name><surname>Chen</surname> <given-names>J. Y. C.</given-names></name> <name><surname>Dong</surname> <given-names>J.</given-names></name> <name><surname>Duffy</surname> <given-names>V. G.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Seven HCI grand challenges</article-title>. <source>Int. J. Hum. Comput. Interact.</source> <volume>35</volume>, <fpage>1229</fpage>&#x02013;<lpage>1269</lpage>. <pub-id pub-id-type="doi">10.1080/10447318.2019.1619259</pub-id></citation>
</ref>
<ref id="B76">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Suresh</surname> <given-names>H.</given-names></name> <name><surname>Gomez</surname> <given-names>S. R.</given-names></name> <name><surname>Nam</surname> <given-names>K. K.</given-names></name> <name><surname>Satyanarayan</surname> <given-names>A.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Beyond expertise and roles: a framework to characterize the stakeholders of interpretable machine learning and their needs,&#x0201D;</article-title> in <source>Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>16</lpage>.</citation>
</ref>
<ref id="B77">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Szymanski</surname> <given-names>M.</given-names></name> <name><surname>Millecamp</surname> <given-names>M.</given-names></name> <name><surname>Verbert</surname> <given-names>K.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;Visual, textual or hybrid: the effect of user expertise on different explanations,&#x0201D;</article-title> in <source>26th International Conference on Intelligent User Interfaces, IUI &#x00027;21</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>109</fpage>&#x02013;<lpage>119</lpage>.</citation>
</ref>
<ref id="B78">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Tversky</surname> <given-names>A.</given-names></name> <name><surname>Kahneman</surname> <given-names>D.</given-names></name></person-group> (<year>1974</year>). <article-title>Judgment under uncertainty: heuristics and biases</article-title>. <source>Science</source> <volume>185</volume>, <fpage>1124</fpage>&#x02013;<lpage>1131</lpage>.<pub-id pub-id-type="pmid">17835457</pub-id></citation></ref>
<ref id="B79">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Tversky</surname> <given-names>A.</given-names></name> <name><surname>Kahneman</surname> <given-names>D.</given-names></name></person-group> (<year>1981</year>). <article-title>The framing of decisions and the psychology of choice</article-title>. <source>Science</source> <volume>211</volume>, <fpage>453</fpage>&#x02013;<lpage>458</lpage>.<pub-id pub-id-type="pmid">7455683</pub-id></citation></ref>
<ref id="B80">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Tversky</surname> <given-names>A.</given-names></name> <name><surname>Kahneman</surname> <given-names>D.</given-names></name></person-group> (<year>1991</year>). <article-title>Loss aversion in riskless choice: a reference-dependent model</article-title>. <source>Q. J. Econ.</source> <volume>106</volume>, <fpage>1039</fpage>&#x02013;<lpage>1061</lpage>.</citation>
</ref>
<ref id="B81">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Vereschak</surname> <given-names>O.</given-names></name> <name><surname>Bailly</surname> <given-names>G.</given-names></name> <name><surname>Caramiaux</surname> <given-names>B.</given-names></name></person-group> (<year>2021</year>). <article-title>&#x0201C;How to evaluate trust in AI-assisted decision making? A survey of empirical methodologies,&#x0201D;</article-title> in <source>Proceedings of the ACM on Human-Computer Interaction</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>39</lpage>.</citation>
</ref>
<ref id="B82">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wachter</surname> <given-names>S.</given-names></name> <name><surname>Mittelstadt</surname> <given-names>B.</given-names></name> <name><surname>Russell</surname> <given-names>C.</given-names></name></person-group> (<year>2018</year>). <article-title>Counterfactual explanations without opening the black box: automated decisions and the GDPR</article-title>. <source>Harvard J. Law Technol.</source> <volume>31</volume>, <fpage>841</fpage>&#x02013;<lpage>887</lpage>. <pub-id pub-id-type="doi">10.2139/ssrn.3063289</pub-id></citation>
</ref>
<ref id="B83">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Wang</surname> <given-names>D.</given-names></name> <name><surname>Yang</surname> <given-names>Q.</given-names></name> <name><surname>Abdul</surname> <given-names>A.</given-names></name> <name><surname>Lim</surname> <given-names>B. Y.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;Designing theory-driven user-centric explainable AI,&#x0201D;</article-title> in <source>Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>15</lpage>.</citation>
</ref>
<ref id="B84">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wason</surname> <given-names>P. C.</given-names></name></person-group> (<year>1960</year>). <article-title>On the failure to eliminate hypotheses in a conceptual task</article-title>. <source>Q. J. Exp. Psychol.</source> <volume>12</volume>, <fpage>129</fpage>&#x02013;<lpage>140</lpage>.</citation>
</ref>
<ref id="B85">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Wischnewski</surname> <given-names>M.</given-names></name> <name><surname>Kr&#x000E4;mer</surname> <given-names>N.</given-names></name> <name><surname>M&#x000FC;ller</surname> <given-names>E.</given-names></name></person-group> (<year>2023</year>). <article-title>&#x0201C;Measuring and understanding trust calibrations for automated systems: a survey of the state-of-the-art and future directions,&#x0201D;</article-title> in <source>Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI &#x00027;23</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>Association for Computing Machinery</publisher-name>).</citation>
</ref>
<ref id="B86">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Yin</surname> <given-names>M.</given-names></name> <name><surname>Wortman Vaughan</surname> <given-names>J.</given-names></name> <name><surname>Wallach</surname> <given-names>H.</given-names></name></person-group> (<year>2019</year>). <article-title>&#x0201C;Understanding the effect of accuracy on trust in machine learning models,&#x0201D;</article-title> in <source>Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>1</fpage>&#x02013;<lpage>12</lpage>.</citation>
</ref>
<ref id="B87">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Yu</surname> <given-names>K.</given-names></name> <name><surname>Berkovsky</surname> <given-names>S.</given-names></name> <name><surname>Taib</surname> <given-names>R.</given-names></name> <name><surname>Conway</surname> <given-names>D.</given-names></name> <name><surname>Zhou</surname> <given-names>J.</given-names></name> <name><surname>Chen</surname> <given-names>F.</given-names></name></person-group> (<year>2017</year>). <article-title>&#x0201C;User trust dynamics: an investigation driven by differences in system performance,&#x0201D;</article-title> in <source>Proceedings of the 22nd International Conference on Intelligent User Interfaces, IUI &#x00027;17</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>307</fpage>&#x02013;<lpage>317</lpage>.</citation>
</ref>
<ref id="B88">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Zhang</surname> <given-names>Y.</given-names></name> <name><surname>Liao</surname> <given-names>Q. V.</given-names></name> <name><surname>Bellamy</surname> <given-names>R. K. E.</given-names></name></person-group> (<year>2020</year>). <article-title>&#x0201C;Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making,&#x0201D;</article-title> in <source>Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* &#x00027;20</source> (<publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>), <fpage>295</fpage>&#x02013;<lpage>305</lpage>.</citation>
</ref>
</ref-list>


</back>
</article> 