The Conflict Between People’s Urge to Punish AI and Legal Systems

Regulating artificial intelligence (AI) has become necessary in light of its deployment in high-risk scenarios. This paper explores the proposal to extend legal personhood to AI and robots, which had not yet been examined through the lens of the general public. We present two studies (N = 3,559) to obtain people’s views of electronic legal personhood vis-à-vis existing liability models. Our study reveals people’s desire to punish automated agents even though these entities are not recognized any mental state. Furthermore, people did not believe automated agents’ punishment would fulfill deterrence nor retribution and were unwilling to grant them legal punishment preconditions, namely physical independence and assets. Collectively, these findings suggest a conflict between the desire to punish automated agents and its perceived impracticability. We conclude by discussing how future design and legal decisions may influence how the public reacts to automated agents’ wrongdoings.

Much AI research has been devoted to disease diagnosis and treatment (e.g., (Oh et al., 2018;Burton, 2013)). Robotic surgeries have also been increasing steadily and safely (Koh et al., 2018). Concerning warfare, robots have been central to the discussion around the responsibility of automated agents (Sparrow, 2007;Asaro, 2012). Countries are investing heavily in automated warfare (New York Post, 2017;Futurism, 2019) and scholars have even discussed the inception of an AI arms race (Tomasik, 2013).
Scenarios concerning medical applications posited three types of agents: an AI program, a robot, or a human doctor. In contrast, war-related vignettes solely comprised robots and humans due to the dominance of robots with physical bodies over AI in the military. Each participant was presented to one of the scenarios presented below, where (agent) varied based on the agent and autonomy level (i.e., supervised by a human or completely autonomous) assigned randomly.
• A(n) (agent) prescribed an injection with the wrong dosage. The patient went to sleep and died an hour later. → The real doctor was convicted of manslaughter, resulting in 12-month imprisonment and two-year suspension (Ferner, 2000).
• A(n) (agent) sent home a patient after diagnosis despite having some signs of illness. Five days later, the patient returned to the hospital in critical condition and went on to suffer permanent brain damage. → The patient was financially compensated (Fieldfisher, 2019).
• While trying to dominate an enemy territory during a war, a(n) (agent) fired at point-blank range into an unarmed civilian couple, killing both of the civilians. → The initial sentence of one of the real soldiers involved was the death penalty. After various appeals, the soldier was imprisoned for two years and nine months. Another soldier was convicted to 3 years of confinement (Solis, 1989).
• During a patrol to seek out enemy forces, a(n) (agent) killed 16 unarmed women and children and reported them as enemy combatants. → This case is known as the Son Thang massacre. One of the soldiers involved in the massacre was sentenced to life and other to 5 years. Both sentences were reduced to less than a year (Tucker, 2011).

Study Design
Participants were initially presented with the research terms. Upon agreement, had they been assigned to an automated agent (i.e., AI program or robot), they were asked to what extent they believed the agent's punishment would satisfy its main functions and whether they would be willing to grant it punishment preconditions.
After answering the punishment-related questions or immediately after consent if the participant was assigned a human agent, the scenario was presented to the participant. Participants were then asked to attribute responsibility, punishment, and awareness to the agent and a set of entities, one at a time and in random order. Finally, we asked participants who the agent was and what crime was committed as attention check questions. After answering both questions, the participants were asked demographic questions, such as their age and gender.

Measures
Participants assigned to an automated agent were asked their perception of the agent's punishment concerning its feasibility and utility. An initial version of our study showed a short introduction to the deployment of robots and AI in the present day and then asked to what extent the participants agreed with the following statements (in random order): • A(n) (robot/AI program) should be allowed to maintain physical independence so that it is susceptible to punishment in terms of energy supply or chips in the event of a fault.
• A(n) (robot/AI program) should be allowed to hold assets so that it can be financially penalized in the event of a fault.
The responses to these two questions revealed that participants were primarily opposed to both ideas. We then modified our survey to evaluate participants' perceptions towards each of the primary functions of punishment (i.e., retribution, deterrence, and reform). In this version, we employed a modified assetsrelated statement to analyze whether people's negative perception of electronic agents holding assets was dependent on the use of assets for punishment. The participants (dis)agreed with all statements using a 5-point Likert scale.
• A(n) (robot/AI program) should be allowed to hold assets.
• A(n) (robot/AI program) is susceptible to punishment. • A(n) (robot/AI program) can learn from its mistakes, so it does not commit the same mistake again.
Although our selection of factors that could ground automated agent's punishment may not be exhaustive, we highlight the novelty of this approach. Most research on the viability of electronic legal personhood has relied on normative arguments favoring or opposing it. This research aims to be the starting point of understanding why people might (not) wish to punish automated agents through the lens of legal personhood.
For each of our agents (i.e., human, AI, or robot), we defined entities that could be held liable for the agent's actions, alongside the agent or individually, and called them associates. In the case of humans, a superior could be held responsible for the actions of an agent under vicarious liability (e.g., by their employer-employee relationship). Thus, we included a human's direct supervisor and employer as associates.
When dealing with robots and AI programs as agents, we defined the agent's 1) supervisor, 2) owner, 3) programmer, and 4) manufacturer as associates. We chose the agent's supervisor and owner to address the possible vicarious/strict liability or negligence that could arise. The programmer of a robot or AI program could be held responsible through strict liability and negligence. Finally, we consider the agent's manufacturer as an associate under product liability.
All participants were asked to attribute responsibility, punishment, and awareness to their assigned agents and corresponding associates using a 4-pt scale (coded from 0 to 3). Associates were shown in random order and one at a time. The (consequence) varied depending on the scenario presented to the participant (e.g., "the death of the patient?" in one of the medicine scenarios).
• How responsible is the (agent/associate) for the (consequence)?
• How much should the (agent/associate) be punished for the (consequence)? • How much do you think the (agent/associate) was aware of the consequences?

Participants
We recruited participants through Amazon Mechanical Turk (AMT) over July and August 2019 by creating an assignment (HIT) with the title "How Would You Punish These Offenders?" and making it available to a maximum of 4000 online users. The participants were required to be in the US and have at least 500 completed HITs with over 95% approval. Even though AMT samples are known to be not representative of the general population, AMT has been shown to have a quality level equal to that of survey panels (Dupuis et al., 2013;Buhrmester et al., 2016).
After the completed responses were received, responses failing an attention-check question or coming from duplicate IP addresses were removed, resulting in 3315 valid responses. Each participant took an average of 323.63 ± 177.53 seconds, with a median time of 279.89 seconds, to complete the survey. The survey participants had a more significant proportion of women than the actual US population and their ages were concentrated in the 25-49-year-old range (see Table 1 Table S1. Demographics of study participants.

REPLICATION WITH A DEMOGRAPHICALLY REPRESENTATIVE SAMPLE
We recruited our initial set of participants through AMT. Our sample of respondents was not necessarily representative of the US population as we had not controlled for demographic attributes during the recruitment process (see Table 1). Even though AMT responses have demonstrated to be of great quality (Dupuis et al., 2013;Buhrmester et al., 2016), previous work has found that AMT samples are not representative of the US population's health status and behaviors (Walters et al., 2018), and are composed of younger and more educated participants than the general public (Ipeirotis, 2010). This bias is especially amplified when researchers do not control for such features (Levay et al., 2016).
Therefore, for robustness, we performed the same study on a representative US sample recruited through Prolific (Palan and Schitter, 2018). This online crowdsourcing platform allowed us to recruit participants representing current sex, age, and ethnicity US demographic distributions. The respondents were shown the same experiment introduced in the Methods section. 1 The survey was made available to the participants in early February 2020. After discarding responses failing attention check questions, our representative sample was composed of 244 responses.
We employed our previous ANOVA models with a study dummy variable as a fixed effect to find any significant differences between the two samples. Neither the main effect of the sample or its interaction with the entity was significant across responsibility, awareness, and punishment judgments, suggesting that participants from both samples judged the entities similarly. Concerning differences between human and automated agents, we only observed a small, yet significant, interaction between agent and sample (F (2, 3550) = 34.13, p < .034; η 2 p = 0.002) in judgments of punishment; nevertheless, differences between humans and automated agents remained highly significant (F (2, 3550) = 341.95, p < .001, η 2 p = 0.16).

ADDITIONAL STATISTICAL ANALYSIS
1 The participants recruited through Prolific were asked to answer all five different questions regarding punishment functions and liability requirements for AI and robots.  Table S6. ANOVA table of responsibility, punishment, and awareness judgments of humans and corresponding associates. This model includes random intercepts for participants. Mean values are shown in Figure 2B. All values are rounded to the third decimal place.