The group cooperation of agents is an important topic studied in the context of autonomous systems (Lee et al., 2018; Hu et al., 2020). Because it is likely that each agent will have individual biases in its actuator or sensor performance, it is an important autonomous ability to analyze these inherent biases and revise the control plan appropriately to continue the group mission. Such biases dynamically change over time during missions, occasionally leading to failures in some functions of an agent. When such changes occur, it is essential to promptly revise the transportation plan using methods such as reinforcement learning. However, this requires constructing a virtual environment that accurately reflects real-world conditions. Therefore, to properly update the operational plan using reinforcement learning, it is necessary to identify the causes of the biases, including any failures, in each agent as they occur.

Previous research on fault diagnosis methods in swarm robotic systems includes the work by O'Keeffe et al. (O’Keeffe et al., 2018). This approach adopts a fault diagnosis mechanism, inspired by biological immune systems, that learns from past diagnostic results to efficiently identify malfunctions based on the behavior of robots. However, the diagnostic tests assumed here only target predictable faults and may struggle when multiple faults occur simultaneously. This difficulty in diagnosis is an inevitable challenge in the advancement of robotic development.

One of the factors complicating this diagnosis is the difficulty in distinguishing the causes of faults.

Suppose that a command base, which controls a group of agents via each command g j (Figure 1A), has detected an anomaly in the position of an agent (e.g., no change in the position was observed). There are two possible causes for the observed anomaly: (1) actuator failures (agent is unable to move) or (2) sensor failures (agent can move, but the move is not captured by the sensor) (Figure 1B). Depending on the hypothesis [the failure may have occurred in the actuators ( h a ) or sensors ( h s ) ], the plan is subsequently calibrated and updated accordingly. However, it is generally difficult to identify which problem caused the anomaly solely through communication between the base and the agent. An intuitive method to identify the correct hypothesis is to execute a collision to the failure agent by other agents to check whether any displacement is observed by the sensor. Such a collision should demonstrate agent displacement; sensor failure would not detect that displacement. Thus, the correct hypothesis can be identified by “planning a group motion.” The question then arises as to whether such planning can be set up autonomously as a “strategy to acquire environmental information” (Friston, 2010).

Such autonomous planning appears to be feasible given the following value function. Suppose that the command g = g 1 , g 2 , ⋯   is issued from the control base, directing the agent’s action to specify which of the hypotheses ( h a , h s ) is supported (Figure 1A). This command updates the agent state to R → R ̃ g . The updated state R ̃ should be denoted as R ̃ h l g because it depends on the hypothesis about the state before the update ( l = a , s ) . As the expected results differ for different hypotheses, the following expression can be used to evaluate the distinction: D = R ̃ h s − R ̃ h a . To ensure appropriate planning g that involves collisions between agents, a non-zero difference D is obtained, and the likelihood of each hypothesis can be determined. We must, therefore, formulate a plan that maximizes D = D g to ensure a significant difference. Accordingly, an autonomous action plan can be formulated to maximize D g as a value function.

However, this maximization task is difficult to complete via conventional gradient-based optimization. Owing to the wide range of possibilities for g , interactions such as collisions are rare events, and for most of the planning phase g , D g = 0 , it is impossible to distinguish between hypotheses. Namely, sub-spaces with finite D are sparse in the overall state space (sparse rewards). In such cases, gradient-based optimization is insufficient for the task of formulating appropriate action plans because the zero-gradient encompasses the vast majority of the space. For such sparse reward optimization, reinforcement learning, which has been thoroughly investigated in the applications of autonomous systems (Huang et al., 2005; Xia and El Kamel, 2016; Zhu et al., 2018; Hu et al., 2020), can be used as an effective alternative.

Reinforcement learning (Nachum et al., 2018; Sutton and Barto, 2018; Barto, 2002) is becoming an established field in the wider context of robotics and system controls (Peng et al., 2018; Finn and Levine (2017). Methodological improvements have been studied intensively, especially by verifications on gaming platforms (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019). Thus, the topic addressed in this study is becoming a subfield known as multi-agent reinforcement learning (MARL) (Busoniu et al., 2006; Gupta et al., 2017; Straub et al., 2020; Bihl et al., 2022; Gronauer and Diepold, 2021). Specific examples of multi-agent missions include unmanned aerial vehicles (UAV) (Bihl et al., 2022; Straub et al., 2020) and sensor resource management (SRM) Malhotra et al., 2017, 1997; Hero and Cochran, 2011; Bihl et al., 2022). The objective of this study can also be regarded as the problem of handling non-stationary environments in multi-agent reinforcement learning (Nguyen et al., 2020; Foerster et al., 2017). As a consequence of failure, agents are vulnerable to the gradual loss of homogeneity. Prior studies have addressed the problem of heterogeneity in multi-agent reinforcement learning (Busoniu et al., 2006; Calvo and Dusparic, 2018; Bihl et al., 2022; Straub et al., 2020; Gronauer and Diepold, 2021). The problem of sparse rewards has also been recognized and discussed as one of the current challenges in reinforcement learning (Wang and Taylor, 2017; Bihl et al., 2022). Recent advancements in reinforcement learning have introduced various innovative methods for handling single-agent or multi-agent scenarios. These approaches have focused on improving sample efficiency, computational costs, and learning stability across different frameworks. One such method is TD7, which utilizes state-action learned embeddings (SALE) for jointly learning embeddings of both states and actions Fujimoto et al., 2024). CrossQ is another approach that improves sample efficiency while significantly reducing computational costs by utilizing batch normalization (Bhatt et al., 2019). Continuous dynamic policy programming (CDPP) extends relative entropy regularized reinforcement learning from value function-based frameworks to actor-critic structures in continuous action spaces, achieving improved sample efficiency and learning stability (Shang et al., 2023). Furthermore, dropout Q-functions (DroQ) employs a small ensemble of dropout Q-functions to enhance computational efficiency while maintaining sample efficiency comparable to randomized ensembled double Q-learning (REDQ) (Hiraoka et al., 2021). In the realm of multi-agent reinforcement learning, multi-agent continuous dynamic policy gradient (MACDPP) has achieved high learning capability and sample efficiency by introducing relative entropy regularization to the centralized training with decentralized execution (CTDE) framework Miao et al., 2024).

The discussion thus far can be generalized as follows: Consider a scenario involving N agents where some anomalies occur, and multiple hypotheses are conceivable. For instance, similar to the earlier example, there could be cases where only a sensor or only an actuator fails in a single agent. Alternatively, there could be scenarios involving multiple agents where anomalies occur in several sensors and actuators, among other various cases. Furthermore, let R denote the state of these N agents, which could be a vector obtained by concatenating the coordinates of N robots. Under hypothesis l , the state R is updated by a command g to a new state R ̃ h l . The difference between the states under hypotheses l and l ′ can be expressed as D < l , l ′ > = R ̃ h l − R ̃ h l ′ , similar to earlier. If a virtual environment that faithfully reproduces these agents’ behavior is prepared, and g that maximizes D < l , l ′ > can be found through reinforcement learning, executing g in real systems and observing the outcomes would allow for discrimination between hypotheses. To search for a g that simultaneously discriminates all hypotheses, reinforcement learning should be conducted to maximize the sum of D < l , l ′ > across all combinations of hypotheses.

As a prototype of such a problem, we considered a system composed of three agents moving on an ( x , y ) -plane, administrated by a command base to perform a cooperative task (Figure 2). In performing the task, each agent is asked to convey an item to a goal post individually. The second agent (#2) is assumed to be unable to move along the y -direction due to an actuator failure. By quickly verifying tiny displacements in each agent, the command base can detect the problem occurring in #2. However, it cannot attribute the cause to either the actuators or the sensors. Consequently, the control base sets hypotheses h a and h s and begins planning the best cooperative motions g * to classify the correct hypothesis via reinforcement learning.

Remarkably, the optimal action plan generated by reinforcement learning showed a human-like solution to pinpoint the problem by colliding other agents with the failed agent. By inducing a collision, the base could identify that #2 is experiencing problems with its actuators rather than sensors. The base then starts planning group motions to complete the conveying task, considering the limited functionality of #2. We observe that the cooperative tasks are facilitated by a learning process wherein other agents appear to compensate for the deficiency of #2 by pushing it toward the goal. In the present study, we employed a simple prototype system to demonstrate that reinforcement learning is extremely effective in setting up a verification plan that pinpoints multiple hypotheses for general cases of system failure.

Let the state space for the agents be R . For instance, given three agents ( j = 1,2,3 ) situated on a x y -plane at positions x j , y j , their states can be specified as R = x 1 , y 1 , x 2 , y 2 , x 3 , y 3 ; that is, points in six-dimensional space. The state is driven by a command g according to the operation plan generated in the command base. When g is assigned to a given R , the state is updated depending on which hypothesis h l is taken, each of which restricts R by individual constraint: g : R → R ̃ h l g , R . The difference D g , R = ∑ < l , l ′ > R ̃ h l g , R − R ̃ h l ′ g , R can then be the measure to evaluate performance and thereby distinguish between the hypotheses. Therefore, the best operation plan for the distinction should be determined as g * = arg max g D g , R .

The naive idea of performing optimization using gradient-based methods is insufficient, owing to the sparseness described in the introduction. For g , D ( g , R ) = 0 , the gradient is zero for most of R because it is incapable of selecting the next update. Accordingly, we employed reinforcement learning as an alternative optimization approach.

Suppose we can evaluate the reward ρ R , g for the action g to be taken for a given state R . In reinforcement learning, decisions aim to maximize the action value Q R , g , rather than maximizing the immediate reward ρ R , g . Although the reward ρ R , g indicates the benefit obtained at that moment, the action value Q R , g represents the benefit accumulated over the future. The governing equation that links the given ρ R , g with Q R , g is known as the Bellman equation, being expressed in self-consistent manner. Users specify the reward function ρ R , g and the detailed specifications of the Bellman equation to self-consistently determine the action value Q R , g using a library. In this study, we employed the OpenAI Gym (Brockman et al., 2016) as such a library. Though the details of the reinforcement learning implementation are found in the general literature, we provide further details using our notation adopted in this paper in Section 1 of the Supplementary Material. In this research, the operational plans are finally determined by the converged action value, Q ̄ R , g , obtained by the self-consistent iterations as g ̄ 0 = arg  max g Q ̄ R 0 , g g ̄ 0 : R 0 → R 1 g ̄ 1 = arg  max g Q ̄ R 1 , g g ̄ 1 : R 1 → R 2 ⋯ . (1)

The workflow required to achieve the mission for the agents, as described in §1, proceeds as follows:

[0a ] To determine if there are errors found in any of the agents, the base issues commands to move all agents by tiny displacements (and consequently, Agent #2 is found to have an error).

The learning process Q α in Step [1] is performed using two virtual spaces V h s , a , corresponding to the two hypotheses: R h l ∈ V h l . Each R h l can take such possibilities under each constraint of its hypothesis (e.g., y 3 cannot be updated due to the actuator error). For an operation g , the state on each virtual space is updated as g : R h s → R ̃ h s g , R h s R h a → R ̃ h a g , R h a . Taking the value function, ρ α g , R h 1 , R h 2 = R ̃ h 1 g , R h 1 − R ̃ h 2 g , R h 1 , (2) the two-fold Q -table is updated self-consistently as Q g , R h 1 , R h 2 = ρ α g , R h 1 , R h 2 + ∑ g ′ , R ′ h 1 , R ′ h 2 F Q g ′ , R ′ h 1 , R ′ h 2 , π g ′ , R ′ h 1 , R ′ h 2 .

Denoting the converged table as Q ̄ α g , R h 1 , R h 2 , the sequence of operations is obtained as given in Equation 1; in other words, g ̄ 0 α , g ̄ 1 α , ⋯ g ̄ M α . (3) The operation sequence generates the two-fold sequence of (virtual) state evolutions as R 1 h s , a → R 2 h s , a → ⋯ → R M h s , a , (4) as shown in Figure 3A.

In Step [2], the agents operate according to the plan expressed by Equation 3 to update (real) states as R 1 → R 2 → ⋯ → R M , (5) to be observed by the command base. The base compares Equations 4, 5 to identify whether h s or h a is the cause of failure ( h a in this case).

In Step [3], Q β -learning is performed for reward ρ β . The reward function ρ β calculates the sum of the individual agents’ rewards, where each agent gets a reward of a / ( r + 1 ) + b ⋅ δ r depending on its distance r from the goal post. Thus, a higher reward is realized when the agent gets closer to the goalpost. By setting a = 0.01 and b = 100.0 , a much higher reward value ( a + b ) is obtained when the agent reaches the goal post ( r = 0 ) . Although learning efficiency varies depending on the values of a and b , a relatively high efficiency was achieved by setting b ≫ a . The operation sequence is then obtained as g ̄ M + 1 β , g ̄ M + 2 β , ⋯ g ̄ L β , by which the states of the agents are updated as R M + 1 → R M + 2 → ⋯ → R L , (6) as shown in Figure 3B.

Figure 3A depicts two-fold trajectories, Equation 4, corresponding to the hypotheses h a and h s . Although R h a = R h s for Agent #1, the branching R h a ≠ R h s occurs for Agent #2 during operations. The branching process earns a score via the value function ρ α in Equation 2, which indicates that the learning Q α was conducted properly. Thus, the ability to capture the difference between h a and h s has been realized. The red dotted circle shown in (a) represents a collision between Agents #2 and #3, inducing the difference between R h a and R h s (the trajectories only reflect the central positions of agents, while each agent has a finite radius similar to its size; therefore, the trajectories themselves do not intersect even when a collision occurs). In addition, the collision strategy is never generated in a rule-based manner, as the agents autonomously deduce their strategy via reinforcement learning.

Three square symbols (closed) situated at the edges of a triangle in Figure 3 represent the goalposts for the conveying mission. Figure 3B shows the real trajectories for the mission, where the initial locations of the agents are the final locations in the panel (a). From their initial locations, Agents #1 and #3 immediately arrived at their goals to complete each mission and subsequently headed to Agent #2 for assistance. Meanwhile, Agent #2 attempted to reach its goal using its limited mobility; that is, only along the x -axis. At the closest position, all three agents coalesced, and Agents #1 and #3 began pushing Agent #2 toward the goal. Though this behavior is simply the consequence of earning more from the value function ρ β , it appears as if Agent #1 wants to assist the malfunctioning agent cooperatively (a video of the behavior shown in Figure 3B is available at the link Hayaschi, 2024). By identifying the constraint h a for the agents in the learning phase Q α , the subsequent learning phase Q β is confirmed to generate the optimal operation plans to ensure that the team maximizes their benefit through cooperative behavior as if an autonomous decision has been made by the team.

During training, if the target reward is not reached in the given number of training sessions, the training process is reset to avoid being trapped by the local solution. In Figure 4, the training curves of rejected trials are shown in blue, whereas the acceptable result is shown in red. Evidently, more learning processes were rejected in Q β (right panel) than in Q α (left panel). This indicates that it is a more challenging task to perform transport planning with three malfunctioning agents than to plan the action to pinpoint a hypothesis between any two. However, under more complex failure conditions, more learning is expected to be rejected for Q α as well, as the number of possible hypotheses increases.

The performance of LSTM and MLP was compared in terms of the success rate for obtaining working trajectories to distinguish between the hypotheses. Notably, even when applying the well-converged Q -table, there is a certain rate required for the non-working trajectories to eliminate the difference between the hypotheses. This is a result of the stochastic nature of the policy in generating the trajectories. In the present work, we took 50 independent Q -tables, each of which was generated from scratch, and obtained 50 corresponding trajectories. The rate required to obtain the trajectories required to distinguish among the hypotheses amounts to 94% for a learning management system (LMS) and 78% for LSTM. In the present comparison, we used the same iteration steps as for Q -table convergence. Because LSTM has a more complex internal structure, its learning quality was expected to be relatively lower than that of an LMS for the common condition, and its performance rate was likewise expected to be lower. In other words, a higher iteration cost is required for LSTM to achieve performance comparable to an LMS. As such, the results shown in the main text are those obtained by the LMS, whereas those obtained by LSTM are presented in the Supplementary Material for reference.

For a simulation in a virtual environment space, we must evaluate the distances between agents at every step. As this is a pairwise evaluation, its computational cost scales as ∼ N 2 for N agents. This cost scaling can be mitigated by using the domain decomposition method wherein each agent is evaluated according to its voxel, and the distance between agents is represented by that between corresponding voxels registered in advance. The corresponding cost scales linearly with N at a much faster rate than the naive ∼ N 2 evaluation method as the number of agents N increases.

Agents performing group missions can suffer from errors during missions. Multiple hypotheses may be devised to explain the causes of such errors. Cooperative behaviors, such as collisions between agents, can be deployed to identify said causes. We considered the autonomous planning of group behaviors via machine-learning techniques. Different hypotheses explaining the causes of the errors lead to different expected states as updated from the same initial state by the same operation. The larger the difference becomes, the better the corresponding operation plan can distinguish between the different hypotheses. In other words, the magnitude of the difference can be the value function to optimize the desired operation plan. Gradient-based optimization does not work well because a tiny fraction among the vast possible operations (e.g., collisions) can capture the difference, leading to a sparse distribution of the finite value for the function. We discovered that reinforcement learning is the obvious choice for such problems. Notably, the optimal plan obtained via reinforcement learning was the operation that causes agents to collide with each other. To identify the causes of error using this plan, we developed a revised mission plan that incorporates the failure of another learning where the malfunctioning agent receives assistance from other agents. By identifying the cause of failure, the reinforcement learning process plans a revised mission plan that considers said failure to ensure an appropriate cooperation procedure. In this study, we conducted tests under the significant constraint that one of the three agents was malfunctioning. As described in §1, the framework can generally be formulated for N agents. Future research will need to explore more detailed studies, including changes in the number of agents and variations in malfunctions. The findings presented in this paper provide initial insights into the capabilities of the proposed methods. Additional comparisons and results based on multiple trials, as well as comparisons with a greater number of baselines, are necessary to substantiate the conclusions of this study further.