Foresight for ethical AI

There is growing expectation that artificial intelligence (AI) developers foresee and mitigate harms that might result from their creations; however, this is exceptionally difficult given the prevalence of emergent behaviors that occur when integrating AI into complex sociotechnical systems. We argue that Naturalistic Decision Making (NDM) principles, models, and tools are well-suited to tackling this challenge. Already applied in high-consequence domains, NDM tools such as the premortem, and others, have been shown to uncover a reasonable set of risks of underlying factors that would lead to ethical harms. Such NDM tools have already been used to develop AI that is more trustworthy and resilient, and can help avoid unintended consequences of AI built with noble intentions. We present predictive policing algorithms as a use case, highlighting various factors that led to ethical harms and how NDM tools could help foresee and mitigate such harms.

. Introduction: ethical AI, policy, and the need for crystal balls Few topics have received as much attention in the last several years as the development of ethical artificial intelligence (AI). There has been an overwhelming development of frameworks and principles-based guidance toward developing ethical AI, although different frameworks use different terms such as ethical, assured, or trustworthy interchangeably (Blasch et al., 2021;Munn, 2022). Such guidance has been developed by government agencies, non-profits, and private companies with the goal of reducing or eliminating ethical harms associated with development and adoption of AI technologies (Hallamaa and Kolliokoski, 2022).
Despite the proliferation of frameworks and guidelines, they can be difficult to put into practice (Munn, 2022). One outcome of this lack of pragmatism in AI ethics is a phenomenon called ethical debt (Heimo and Holvitie, 2020;Petrozzino, 2021). Ethical debt is analogous to technical debt, where AI tools are developed under the presumption that the AI solution itself is ethical. For the sake of development efficiency and lack of concrete means to address ethical consequences, such consequences often go ignored or unmitigated until ethical harms become manifest. A key aspect of ethical AI that we focus on for the remainder of this manuscript is the responsibility of AI developers to foresee and mitigate harms. This concept is epitomized by the Ad Hoc Committee on Responsible Computing's first rule of responsible computing (Miller, 2011, p. 58): "The people who design, develop, or deploy a computing artifact are morally responsible for that artifact, and for the foreseeable effects of that artifact. This responsibility is shared with other people who design, develop, deploy or knowingly use the artifact as part of a sociotechnical system." This need to foresee and mitigate harms is being codified in part via new legislation, mostly in Europe, that makes it easier for individuals to sue AI developers for harms (Hacker, 2022) or requires AI-driven social platforms to manage harms or face various fines and penalties (Husovec and Laguna, 2022). The United States has echoed these sentiments in the new "AI Bill of Rights, " although there are already criticisms of this policy's limitations (Hine and Floridi, 2023).
Despite the legal incentives, foreseeing and mitigating AIdriven harms is easier said than done given the prevalence of emergent behaviors when AI is integrated into sociotechnical systems. Emergent behaviors or properties can be defined as those arising from interactions among components in a sociotechnical system, that are otherwise not observable when investigating any one component (Carayon et al., 2015). Such behaviors exist at the individual level, where introducing AI into a workflow can increase human complacency (Ezer et al., 2019), or drive workflow adaptations based on changes in trust with the AI over time . There is also the issue of complexity at the higher, sociotechnical system level. We have long known that sociotechnical systems will stretch when novel technology is introduced, altering the types of work performed (Woods, 2006;Sheridan, 2008). For example, introducing AI into the domain of intelligence analysis raises questions about the nature of analysts' work changing to focus on inspecting AI outputs vs. conducting analysis, as well as other changes to the nature of workforce collaboration and management (Vogel et al., 2021).
In the AI ethics community, Mittelstadt et al. (2016) have discussed examples of transformative effects where introduction of AI drives humans to re-envision their world, classifying more nuanced phenomena discretely. AI ethicists have acknowledged the need for taking a more holistic view (e.g., Asaro, 2019), arguing for a need to frame ethical discourse on human-AI teams rather than individual agents in simple "trolley problems" that neglect emergent behaviors in complex systems (Borenstein et al., 2019;Cañas, 2022).
These difficulties are echoed in the legal community, where there are concerns about the adequacy of tort, liability, and negligence law to cope with novel AI technology (e.g., see Sullivan and Schweikart, 2019;Selbst, 2020;Stein, 2022). A key issue is that these legal doctrines rely on a reasonable expectation of an entity to foresee harms in order to be held liable for them. That is, if one can argue that an AI developer could not reasonably have been expected to foresee an emergent harm, then there is no legal recourse for those harmed by the AI-effectively nullifying new policies to hold AI developers accountable. Hacker et al. (2020) argue that explainability will be required to overcome such issues. We believe explainability alone is insufficient as it does not address the underlying issue of emergence in sociotechnical systems. Along these lines, Selbst (2020) asserts that individuals cannot reasonably be expected to have the capacity to understand AI, regardless of how explainable or interpretable it is. More directly, Cofone (2018) proposes characterizing AI systems by their level of emergence, or degree to which there is predictability of the AI interacting with the environment.
In summary, new regulatory policies increasing liability for AI developers have merit in theory but may be difficult to exercise in practice. They will provide incentive for AI developers to more carefully consider the ethical harms their technologies might cause, changing the calculus of accruing ethical debt for the better. However, the relevant legal constructs to enforce this feedback loop are based on the concept of what harms one may reasonably expect the developer to foresee. Not only is this difficult in practice because of emergent behaviors associated with AI, but it is reactive, requiring harms to occur before the developer is held accountable. Therefore, a major objective going forward should be to increase foresight of ethical harms created by adding AI into complex sociotechnical systems.
. Increasing understanding and foresight . . Understanding: naturalistic decision making Naturalistic decision making (NDM) is the research tradition that studies decision making processes under realistic conditions (Klein et al., 1993;Klein, 2022). NDM focuses on the sensemaking and decision making processes of experienced people in work environments characterized by complexity, uncertainty, and time pressure. NDM has been contrasted to microcognitive judgement and decision paradigms based on giving naïve participants artificial tasks in controlled laboratory environments, which are not representative of the operational settings in which findings would be applied (Klein et al., 1993). NDM research has shown that experienced decision makers do not simply choose from a set of options, but instead recognize cues and patterns, diagnose situations as anomalous, and evaluate their options by mental simulation in highly novel situations. NDM accounts for the adaptive and integrative relationships among sensemaking and decision making processes (e.g., Ward et al., 2018), and has proven useful where systems developed based on laboratory decision paradigms have failed (Nemeth and Klein, 2010).
NDM encourages a macrocognitive perspective by applying naturalistic inquiry methods to study and support cognitive functions (e.g., sensemaking, planning, and coordinating) and processes (e.g., detecting problems and maintaining common ground), as well as the development of expertise (Hoffman et al., 2014; Naturalistic Decision Making Association (NDMA), n.d.). Given these various aims, NDM is regularly applied to a variety of complex and high-consequence work domains, such as aviation, healthcare, firefighting, intelligence analysis, and military decision making (see Klein and Wright, 2016). Of particular interest here is that the holistic view of macrocognition enables the study of emergent phenomena in sociotechnical systems, such as feedback loops, equilibrium, and self-organization, etc. (Klein et al., 2003). Employment of this macrocognitive perspective has resulted in the development of several tools and methods for improving decisions.

. . Foresight: the premortem and other NDM tools
A premortem-the opposite of a postmortem-is a lightweight method that has been applied in various high-consequence .
domains both inside and outside of the NDM community to proactively determine why a project or plan has hypothetically failed (Klein, 2007(Klein, , 2022Veinott et al., 2010;Eckert, 2015). To begin a premortem, team members are given two minutes to brainstorm reasons for failure (i.e., project risks). A facilitator then asks each person to share a reason from their list, alternating between participants until all unique reasons have been recorded (Klein, 2007). There are some methodological variants that include generating mitigations for the identified risks (Veinott et al., 2010;Klein, 2022), or a more formal elicitation process aimed at uncovering potential root causes of failure (Eckert, 2015).
A key benefit of the premortem lies not in the proactive enumeration of risks, but in the types of risks that are typically identified. With a diverse group of participants, premortems can uncover creative and novel risks that span multiple areas of expertise. Such risks could not be identified by project managers alone (Klein, 2007;Bettin et al., 2022). Further, premortems identify risks at the intersection of humans and technology, which require an interdisciplinary lens to uncover. Bettin et al. (2022) describes how the non-constrained thinking inherent in premortems results in the identification of risks within sociotechnical systems that are often overlooked as they are outside of immediate control, including risks associated with hypothetical user behavior patterns, and those related to broader longitudinal cultural, affective, or motivational factors. This resonates with other work demonstrating that premortems can identify other risks that may not be immediately obvious to a single leader or analyst, including a broad array of personnel-and policybased risks (Klein, 2007), risks stemming from programmatics and team interactions (Eckert, 2015), and risks stemming from operational, organizational, technical, and business factors (Gallop et al., 2016). These outcomes position the premortem as a potentially invaluable method for predicting ethical harms (i.e. manifestations of risks) resulting from introducing AI into complex sociotechnical systems.
While the premortem is a sound candidate for the challenge that we present, it should be noted that it is not the only NDM approach that could be employed to identify AI-driven harms. Recent work has shown that analytic wargames involving experienced players in naturalistic settings can be used to test disruptive technologies and uncover emergent behaviors (de Rosa and De Gloria, 2021), and to explore how the work system may stretch with the introduction of new technologies (Dorton et al., 2020). Other approaches have included the use of naturalistic methods such as the critical incident technique to develop evidence-based checklists for AI developers (e.g., Dorton, 2022). Further still, new methods based on naturalistic inquiry such as Systematic Contributors and Adaptation Diagramming (SCAD; Jefferies et al., 2022) and Joint Activity Monitoring (JAM; Morey et al., 2022) have been developed to attempt to identify issues in work systems more proactively.
. . Predictive policing as a use case "Predictive policing" typically refers to the application of quantitative techniques to direct police intervention to prevent

Relevant risks foreseen via NDM tools
Police were not skilled in data entry and processing Premortems (Bettin et al., 2022) have foreseen risks with assumptions of user skills. NDM research-driven AI checklists (e.g., Dorton, 2022) have also highlighted this issue Data were intentionally manipulated to suppress crime rates Premortems have foreseen risks from bad faith actors (e.g., Veinott et al., 2010) and from the influence of leadership on policies and practices (e.g., Klein, 2007) False veneer of objectivism provided to biased data created a self-fulfilling prediction Wargames (e.g., Dorton et al., 2020) would identify such risks by nature of gameplay No guidance on how to operationalize system outcomes Premortems (Bettin et al., 2022) have foreseen risks of users being unfamiliar with systems and not following protocols. Wargames (e.g., Dorton et al., 2020) would identify such risks by nature of gameplay crimes by predicting one or both of the following: Places and times of crimes, and people who are likely to commit crimes (Degeling and Berendt, 2018;Coeckelbergh, 2020;Miró-Llinares, 2020). Such technologies were developed with noble intentions to reduce crime and increase the objectivity and efficiency of policing. However, there is a growing consensus that AI-driven predictive policing technologies are not only of dubious efficacy but also result in numerous ethical harms such as disproportionate surveillance and targeting of specific socioeconomic or racial groups, and threatening privacy and civil liberties (Degeling and Berendt, 2018;Asaro, 2019;Richardson et al., 2019;Coeckelbergh, 2020;Miró-Llinares, 2020). This example may prompt the questions, "How did this go wrong?" and "Why didn't we see this coming?" The left column in Table 1 provides a non-exhaustive list of various contributing factors to ethical harms in predictive policing (Degeling and Berendt, 2018;Asaro, 2019;Richardson et al., 2019;Miró-Llinares, 2020;Alikhademi et al., 2021). First, police are not trained data scientists, so data may not be correctly and consistently annotated (e.g., approximate or inaccurate time stamps in their reports make models heavily dependent on temporal relationships less reliable). Second, data have been intentionally manipulated in some instances (e.g., asking people not to press charges) to create the appearance of reduced crime rates. Third, AI was fed data that were systematically biased (e.g., from jurisdictions under consent decrees for discriminatory practices), resulting in crime location predictions that simply reflect biases and the deference of judgement to faulty model predictions. Finally, police were not informed about what to do with AI outputs, resulting in inconsistent practices when patrolling AI-designated "high risk" areas.
The right column of Table 1 shows how just a few NDM tools have been shown to proactively identify such risk factors in comparable systems. Of particular interest is that the premortem technique addresses three of the four enumerated (or analogous) factors in this use case, despite there being a relatively sparse body of extant literature on the premortem from which to draw. Therefore, we argue that premortems can proactively identify ethical harms from, or failures of, AI. It seems more likely, however, that premortems and other NDM tools will help AI developers .
foresee the underlying factors that contribute to ethical harms, as shown in this predictive policing example.

. Discussion
Overcoming ethical debt in AI development is a substantial challenge. More specifically, foreseeing and mitigating ethical harms is exceptionally difficult given the prevalence of emergent behaviors that occur when integrating AI into sociotechnical systems. We have argued that the NDM community (as a personification of its people, principles, and tools) is well-suited to tackling this challenge. However, we do caution that the AI ethics community should not view NDM or macrocognitive approaches as a panacea: that is, adding an NDM practitioner to your team or using an NDM method does not guarantee success in foreseeing and mitigating harms.
Premortems do not guarantee an exhaustive set of risks will be identified (Bettin et al., 2022). Similarly, methods such as analytic games (e.g., de Rosa and De Gloria, 2021) and other checklists based on naturalistic inquiry (e.g., Dorton, 2022) are bounded in scope by not only the scenarios and injects examined, but also the expertise of the participants involved in their application. More generally, there is a long-standing challenge of developing tools for an envisioned world with technological change (e.g., Woods and Dekker, 2010). Even newer tools focused on proactive identification of risks such as SCAD (Jefferies et al., 2022) and JAM (Morey et al., 2022) rely on collecting data from existing sociotechnical systems, making them insufficient to truly provide foresight before integrating new AI into work systems. Thus, future methodological research is needed to develop and refine methods to enable greater foresight of ethical harms. For example, could a premortem be scoped or targeted to foresee ethical harms vs. more general project risks?
While we have provided an argument for how NDM tools can increase foresight (i.e., proactively identifying emergent risks), we have not addressed problem solving (i.e., what to do with identified risks). While anecdotal evidence has shown that the outputs of NDM tools such as the premortem can inform requirements, designs, and test and evaluation activities, additional work is needed to more concretely map the outputs of NDM tools like the premortem into the AI development process.
Acknowledging these limitations, we maintain that the use of NDM principles and tools has obvious value to AI developers. This value extends beyond developing ethical AI. NDM approaches have been used to glean insights on developing trustworthy AI (e.g., Dorton andDorton, 2022) and resilient AI technologies (e.g., Neville et al., 2022). In summary, we have attempted to craft our argument while providing different messages to two different audiences. To the AI ethics community: While NDM is not a panacea, the NDM community has a considerable head start addressing the aforementioned challenges. To the NDM community: There is an opportunity to refine our models and tools to improve foresight and reduce ethical debt in AI. So, can NDM tools uncover an exhaustive set of emergent harms for a given AI technology? No. It would be naive to think so. Can NDM tools uncover a reasonable set of emergent harms? We believe so.

Author contributions
SD: conceptualization, investigation, and writing-original draft. LM and BA: writing-original draft. DB: conceptualization and writing-original draft. SD and LM: writing-final draft. All authors contributed to the article and approved the submitted version.

Funding
Funds for Open Access Publication Fees were from the Social & Behavioral Sciences Department [L175] of the MITRE Corporation.