Multiplayer reach-avoid differential games with simple motions: A review

1 Introduction

Multi-robot systems, including self-driving cars and unmanned aerial vehicles, are becoming a topic of great interest. These systems have significant advantages over a single robot because they can share the workload and cooperatively complete complicated tasks, such as automated package delivery, disaster survivors search, infrastructure protection and region patrolling (Chen et al., 2016; Shishika and Kumar, 2018; Shishika and Kumar, 2020; Yan et al., 2022; Yan et al., 2019b; Yan et al., 2020; Shishika et al., 2020; Shishika et al., 2021; Deng et al., 2021; Guerrero-Bonilla et al., 2021; Lee and Bakolas, 2021; Von Moll et al., 2022b). Of particular relevance to this paper is a class of scenarios related to security and cooperation-competition applications. Specifically, we consider multiplayer reach-avoid (M-RA) differential games, in which multiple robots are used to protect a goal region of interest against a group of malicious robots which aim to enter the goal region without being captured.

Compared with the classical pursuit-evasion games in which the capture is the only competition goal, M-RA differential games are more complicated and have more practical significance, as the evaders aim to reach a target set and avoid the capture at the same time. According to the degree of abstraction and physical constraints, the players can be described by different mathematical models, such as simple motion (Isaacs, 1965), Dubins car with the minimum turning radius (Dubins, 1957), and Reeds–Shepp car with the backward move (Reeds and Shepp, 1990). This review focuses on the simple motion, or the first-order integrator with bounded inputs in the language of control theory, in which the player moves with a bounded speed and can change its heading instantaneously. Such a model is a suitable abstraction for mobile robots or robotic vehicles which have speed limitations and high maneuverability, for instance, humanoid robots, quadrotor unmanned aerial vehicle and small underground vehicles, and due to its simplicity, this model has been extensively studied with fruitful results in differential games (Fisac et al., 2015; Chen et al., 2018; Ibragimov et al., 2018; Yan et al., 2020; Fu and Liu, 2021; Yan et al., 2021a; Yan et al., 2021b; Liang et al., 2022; Wang et al., 2022; Yan et al., 2022).

The challenges of solving M-RA differential games with simple motions can be broadly divided into two categories: non-unique terminal conditions, and complex cooperation and competition pattern (Yan et al., 2020; Yan et al., 2022). Non-unique terminal conditions, where the game could end up with either capture or entry into the goal region, largely complicate the strategy synthesis which involves integrating backward trajectories from differential terminal surfaces (Isaacs, 1965). This results from a lack of systematic analysis methods in the presence of complicated singular surfaces occurring in the backward computation. At the inter-agent level, grouping players into two opposing teams is intrinsically accompanied with complex cooperation within team members and goal-driven inter-team competition. For instance, it is not hard to imagine a scenario where cooperation between two pursuers is necessary for winning against an evader while any one of them fails to do so (Yan et al., 2020). Like prey animals, some evaders may lure the pursuers away from the goal region or sacrifice themselves through being captured such that the other evaders successfully reach the goal region.

This review is concerned with the M-RA differential games, with a particular interest in simple motions, which were first discussed by (Mitchell et al., 2005; Margellos and Lygeros, 2011; Zhou et al., 2012) and then extended into many variations and practical applications (Huang et al., 2014; Selvakumar and Bakolas, 2019; Fu and Liu, 2020). The problem is closely related to lifeline games (Garcia et al., 2019b; Yan et al., 2021a; Yan et al., 2021b; Chen and Yu, 2022), two-target differential games (Blaquière et al., 1969; Olsder and Breakwell, 1974; Pachter and Getz, 1980; Getz and Pachter, 1981) and target guarding differential games (Mohanan et al., 2018). Moreover, the problem has high relevance to scenarios involving underground vehicles guarding a building, unmanned aerial vehicles patrolling against illegal poachers and unmanned surface vehicles patrolling around a prohibited water area.

The remainder of this paper is organized as follows. Section 2 introduces the background on simple motion, game elements and core concepts. In Section 3, we review two most common methods in M-RA differential games. We detail the barrier construction in Section 4 for several interesting M-RA differential games. We present an integer linear programming formulation for task allocation in Section 5. We review three classical strategies in Section 6. In Section 7, we discuss the limitations in the literature and possible directions for future research. Finally, Section 8 concludes the paper.

2 Background

M-RA differential games draw concepts from the fields of differential games, reachability, control and robotics. In this section, we first introduce the system dynamics, assumptions and game elements used throughout the rest of the paper in Section 2.1. Then, Section 2.2 contains a representative, but not complete, discussion of the possible applications. We conclude the section with the core concepts in differential games for qualitative and quantitative analysis in Section 2.3.

2.1 Simple motion and game elements

We consider N_p + N_e players partitioned into two teams, a team of N_p pursuers (also called defenders), $\mathcal{P}=\{P_1,\dots ,P_{N_p}\}$, and a team of N_e evaders (also called attackers), $\mathcal{E}=\{E_1,\dots ,E_{N_e}\}$. The players move in an n-dimensional Euclidean open/closed game region $\mathrm{\Omega }\subset {\mathbb{R}}^n(n\ge 2)$ separated by an (n−1)-dimensional hypersurface $\mathcal{T}\subset {\mathbb{R}}^{n-1}$ into two regions: play region Ω_play and goal region Ω_goal, as shown in Figure 1. The players are assumed to be point masses and they have simple motion as Isaacs stated (Isaacs, 1965), i.e., they are holonomic. Let ${\mathit{x}}_{P_i}\in {\mathbb{R}}^n$ and ${\mathit{x}}_{E_j}\in {\mathbb{R}}^n$ be the positions of P_i and E_j, respectively. The dynamics of the players are described by the following differential equations

$\begin{array}{ccccc}\hfill {\dot{\mathit{x}}}_{P_i}& =v_{P_i}{\mathit{u}}_{P_i},\hfill & \hfill {\mathit{x}}_{P_i}\left(0\right)& ={\mathit{x}}_{P_i}^0,\hfill & \hfill P_i\in \mathcal{P},\\ \hfill {\dot{\mathit{x}}}_{E_j}& =v_{E_j}{\mathit{u}}_{E_j},\hfill & \hfill {\mathit{x}}_{E_j}\left(0\right)& ={\mathit{x}}_{E_j}^0,\hfill & \hfill E_j\in \mathcal{E},\end{array}(1)$

where ${\mathit{x}}_{P_i}^0$ and ${\mathit{x}}_{E_j}^0$ are the initial positions of P_i and E_j, and $v_{P_i}\in {\mathbb{R}}_{>0}$ and $v_{E_j}\in {\mathbb{R}}_{>0}$ denote the speed of P_i and E_j, respectively. The control inputs for P_i and E_j are their respective instantaneous headings ${\mathit{u}}_{P_i}$ and ${\mathit{u}}_{E_j}$, which satisfy the constraint $\mathbb{U}=\{\mathit{u}\in {\mathbb{R}}^n\mid \Vert \mathit{u}{\Vert }_2\le 1\}$. The simple motion (1) models the players which have limited moving speeds and can change their headings instantaneously. We make the following assumptions.

FIGURE 1

FIGURE 1. Multiplayer reach-avoid (M-RA) differential games, where multiple evaders (red) aim to enter the goal region, while the pursuers (blue) are tasked to protect the goal region by capturing the evaders (Yan et al., 2020).

2.2 Applications

As the players’ objectives imply, M-RA differential games have high relevance to the adversarial scenarios in which players compete or cooperate for a set of states in the game state space. For example, mobile ground vehicles can be employed to defend a building of interest so as to minimize some metric, such as the number of malicious vehicles entering the building (Fu and Liu, 2020; Shishika and Kumar, 2020; Shishika et al., 2021). In wildlife protection, the use of unmanned aerial vehicles against illegal poachers is a promising alternative to typical field methods. As more and more attacking boats occur in many waterside cities, deploying patrolling boats is a sensible and feasible solution to protecting stationary ferries. In path planning, a group of vehicles aim to get into some goal region or escape from a bounded region through an exit, while avoiding dangerous situations, such as collisions with moving obstacles (Yan et al., 2019b; Yan et al., 2020; Yan et al., 2022).

2.3 Barriers, winning regions and strategies

In general, the problems in M-RA differential games are classified into two categories: game of kind and game of degree. In a game of kind, the goal is, given a winning condition, to determine which team (player) can win the game, and therefore the game solution is win or lose for a team (player). If the game winner is known with the result of the game of kind, the natural question to ask is how to design strategies so as to ensure the winning and optimize some metric simultaneously, for instance, the distance to the goal region from the perspective of the evasion team if the captured cannot be avoided. Technically, such a problem leads to a game of degree, in which the focus is, given a payoff function, to find the (saddle-point) equilibrium strategies for the players.

2.3.1 Barriers and winning regions

In order to solve the game of kind systematically, Isaacs introduced the concept of barrier (Isaacs, 1965), a surface that divides the entire game state space into two disjoint parts: pursuit winning region (PWR) and evasion winning region (EWR). With a particular interest in the case of multiple pursuers against one evader, the PWR is the set of initial states, from which the pursuit team can ensure the capture before the evader enters the goal region. The EWR, complementary to the PWR, is the set of initial states, from which the evader guarantees to reach the goal region regardless. Naturally, constructing the barrier becomes the core of solving a game of kind. Formally, the PWR ${\mathcal{W}}_P$, EWR ${\mathcal{W}}_E$ and barrier $\mathcal{B}$ for $\mathcal{P}$ against E_j are respectively given by

$\begin{array}{cc}\hfill {\mathcal{W}}_P& =\left\{\mathit{x}=\left({\mathit{x}}_{P_1},\dots ,{\mathit{x}}_{P_{N_p}},{\mathit{x}}_{E_j}\right)\mid \exists \mathit{u}\in {\mathrm{\Sigma }}_P,\forall {\mathit{u}}_{E_j}\in \mathbb{U},s.t.,\mathcal{P}\text{\hspace{0.17em}wins\hspace{0.17em}against\hspace{0.17em}}{E}_j\text{\hspace{0.17em}from\hspace{0.17em}}\mathit{x}\right\},\hfill \\ \hfill {\mathcal{W}}_E& =\left\{\mathit{x}=\left({\mathit{x}}_{P_1},\dots ,{\mathit{x}}_{P_{N_p}},{\mathit{x}}_{E_j}\right)\mid \exists {\mathit{u}}_{E_j}\in \mathbb{U},\forall \mathit{u}\in {\mathrm{\Sigma }}_P,s.t.,E_j\text{\hspace{0.17em}wins\hspace{0.17em}against\hspace{0.17em}}\mathcal{P}\text{\hspace{0.17em}from\hspace{0.17em}}\mathit{x}\right\},\hfill \\ \hfill \mathcal{B}& =\left\{\mathit{x}=\left({\mathit{x}}_{P_1},\dots ,{\mathit{x}}_{P_{N_p}},{\mathit{x}}_{E_j}\right)\mid \exists \mathit{u}\in {\mathrm{\Sigma }}_P,\exists {\mathit{u}}_{E_j}\in \mathbb{U},s.t.,\mathcal{P}\text{\hspace{0.17em}and\hspace{0.17em}}{E}_j\text{\hspace{0.17em}cannot\hspace{0.17em}win\hspace{0.17em}from\hspace{0.17em}}\mathit{x}\right\},\hfill \end{array}$

which can be also described by fixing the pursuers/evaders’ positions. Due to the usefulness of knowing the game winner before the game actually runs, huge progress has been made on the study of barriers (Yan et al., 2017; Shishika and Kumar, 2018; Yan et al., 2019a; Shishika et al., 2020; Yan et al., 2020; Liang et al., 2022; Lee and Bakolas, 2021; Yan et al., 2021a; Yan et al., 2021b; Von Moll et al., 2022b; Chen and Yu, 2022).

2.3.2 Strategies

Regarding the game of degree, the strategy type has a huge impact on the approaches of seeking equilibrium strategies and the inherent computational complexity. In a nutshell, a strategy (policy) of a player resolves the choices in each game state based on its available information at the moment. There are four basic types of strategies for the players in differential games–open loop, state feedback, non-anticipative and anticipative strategies (Mitchell et al., 2005). An open loop strategy requires that each player decides its entire controls u(τ) for all $\tau \in \left[t,\infty \right)$ without any knowledge of the other players’ decisions. A state feedback strategy allows each player to choose u(τ) based on the current value of the state. A non-anticipative strategy allows a player (team) to choose u(τ) with all the information of state feedback, plus the other players’ current input. While the other players are at a slight disadvantage under this strategy structure, at a minimum they have access to using state feedback, because the player must declare its strategy before the other players choose a specific input and thus the other players can determine the response of the player to any input signal. An anticipative strategy would be equivalent to allowing a player to choose u(τ) based on knowledge of all future inputs of the other players; in other words, the other players would have to reveal their entire input signals in advance to this player.

3 Methods

We begin our discussion by reviewing the two most common methods, geometric method and Hamilton-Jacobi-Isaacs (HJI) method, that are widely used in M-RA differential games with simple motions, to solve the induced games of kind and games of degree. The geometric method leverages the player dynamics, i.e., simple motion, under which the optimal trajectory of the player is a straight line in many cases (Isaacs, 1965; Yan et al., 2019b; Yan et al., 2020; Yan et al., 2022). The HJI method is more general and is able to handle with more complicated player dynamics. However, it also suffers from high computational complexity Mitchell et al. (2005); Margellos and Lygeros (2011); Chen et al. (2018); Fisac et al. (2015).

3.1 Geometric method

If the optimal trajectories of the players are known to be composed of straight lines, which is common under the simple motion, solving the game is closely related to constructing the dominance regions (Isaacs, 1965; Oyler et al., 2016), where a point in the game region is said to be dominated by one of the players if that player is able to reach the point before the other players, regardless of the other players’ best effort (the capture radius is also taken into account). A dominance region is then the set of all points dominated by a particular player. We first introduce two classical and predominant dominance regions: Voronoi cell and Apollonius circle, and then present a more general function-based dominance region.

FIGURE 2

FIGURE 2. Dominance regions for multiple pursuers against one evader: Voronoi cell (A), Apollonius circle (B) and function-based (C), where the crosses are the centers of the Apollonius circles.

3.2 Hamilton-Jacobi-Isaacs method

Let $\mathit{x}=({\mathit{x}}_{P_1},\dots ,{\mathit{x}}_{P_{N_p}},{\mathit{x}}_{E_1},\dots ,{\mathit{x}}_{E_{N_e}})\in {\mathbb{R}}^{n(N_p+N_e)}$ be the state of the game, and the control inputs of two teams are denoted as ${\mathit{u}}_p=({\mathit{u}}_{P_1},\dots ,{\mathit{u}}_{P_{N_p}})\in {\mathbb{R}}^{n{N}_p}$ and ${\mathit{u}}_e=({\mathit{u}}_{E_1},\dots ,{\mathit{u}}_{E_{N_e}})\in {\mathbb{R}}^{n{N}_e}$. Consider an M-RA differential game with the dynamics (1), and the terminal set and the terminal payoff respectively are as follows

$\mathcal{M}=\left\{\mathit{x}\mid g\left(\mathit{x}\right)\le 0\right\},J=\mathrm{\Phi }\left(\mathit{x}\left(t_f\right)\right),\mathit{x}\left(t_f\right)\in \mathcal{M}.(3)$

Since the M-RA differential game is zero-sum in general, the corresponding value function V(x) is the unique viscosity solution to the HJI equation

$\underset{{\mathit{u}}_e}{\min }\underset{{\mathit{u}}_p}{\max }H\left(\mathit{x},\mathit{\lambda },{\mathit{u}}_p,{\mathit{u}}_e\right)=\underset{{\mathit{u}}_p}{\max }\underset{{\mathit{u}}_e}{\min }H\left(\mathit{x},\mathit{\lambda },{\mathit{u}}_p,{\mathit{u}}_e\right)=\sum _{i=1}^{N_p}{v}_{P_i}\Vert {\mathit{\lambda }}_{P_i}{\Vert }_2-\sum _{j=1}^{N_e}{v}_{E_j}\Vert {\mathit{\lambda }}_{E_j}{\Vert }_2=0,(4)$

where the Hamiltonian is defined as $H\left(\mathit{x},\mathit{\lambda },{\mathit{u}}_p,{\mathit{u}}_e\right)={\mathit{\lambda }}^{\top }\mathit{f}$, f is the stacked dynamics (1), and $\mathit{\lambda }=({\mathit{\lambda }}_{P_1},\dots ,{\mathit{\lambda }}_{P_{N_p}},{\mathit{\lambda }}_{E_1},\dots ,{\mathit{\lambda }}_{E_{N_e}})\in {\mathbb{R}}^{n(N_p+N_e)}$ is the costate whose value equals to the gradient of V(x), i.e., λ = ∇V(x), and the underlying minimax controls are ${\mathit{u}}_{P_i}^{*}=\frac{{\mathit{\lambda }}_{P_i}}{\Vert {\mathit{\lambda }}_{P_i}{\Vert }_2}$ and ${\mathit{u}}_{E_j}^{*}=-\frac{{\mathit{\lambda }}_{E_j}}{\Vert {\mathit{\lambda }}_{E_j}{\Vert }_2}$. The boundary values for the HJI equation satisfy V(x) = Φ(x) for all $\mathit{x}\in \mathcal{M}$. Then, the method of characteristics can be used to solve (4), which originally is a partial differential equation (PDE) and then converted into a system of Euler-Lagrange (EL) ordinary differential equations (ELODEs).

More specifically, we define the minimax Hamiltonian as

$H^{*}\left(\mathit{x},\mathit{\lambda }\right)=\underset{{\mathit{u}}_e}{\min }\underset{{\mathit{u}}_p}{\max }H\left(\mathit{x},\mathit{\lambda },{\mathit{u}}_p,{\mathit{u}}_e\right)=\sum _{i=1}^{N_p}{v}_{P_i}\Vert {\mathit{\lambda }}_{P_i}{\Vert }_2-\sum _{j=1}^{N_e}{v}_{E_j}\Vert {\mathit{\lambda }}_{E_j}{\Vert }_2.(5)$

If V(x) is twice continuously differentiable, the equilibrium trajectories are determined by the following ELODE:

$\dot{\mathit{x}}=\frac{\partial H^{*}\left(\mathit{x},\mathit{\lambda }\right)}{\partial \mathit{\lambda }}={\mathit{f}}^{*},\dot{\mathit{\lambda }}=-\frac{\partial H^{*}\left(\mathit{x},\mathit{\lambda }\right)}{\partial \mathit{x}}=0,(6)$

where f* is the stacked dynamics f under the minimax controls. Such equilibrium trajectories are called regular equilibrium trajectories and the corresponding optimal controls are called regular equilibrium controls. The ELODE (6) reveals that for M-RA differential games with simple-motion players, the regular equilibrium trajectories are straight lines and the regular equilibrium controls are constant, which validates geometric methods in such games. Along the regular equilibrium trajectories, it holds that

$\dot{V}\left(\mathit{x}\right)={\nabla }^{\top }V\left(\mathit{x}\right)\dot{\mathit{x}}={\mathit{\lambda }}^{\top }{\mathit{f}}^{*}=0,$

implying that the value function is constant.

The HJI method solves an M-RA differential game by integrating the ODE system (6) in inverse time initially from the boundary of $\mathcal{M}$. At a point $\mathit{x}\in \partial \mathcal{M}$, the costate satisfies

$\mathit{\lambda }=\nabla \mathrm{\Phi }\left(\mathit{x}\right)+\mu \nabla g\left(\mathit{x}\right),(7)$

where $\mu \in \mathbb{R}$ is the Lagrange multiplier which can be determined by substituting (7) into (4). As soon as the costate on the boundary of $\mathcal{M}$ is obtained, one can solve the ODE system (6) to get the value function and equilibrium controls along the regular equilibrium trajectories. This method is widely used in the study of the problem of active target defense [Pachter et al. (2019); Garcia et al. (2018); Liang et al. (2019); Garcia et al. (2019a); Liang et al. (2021)], where the target is a maneuvering player that cooperates with a defender against an attacker. In Akilan and Fuchs (2017); Von Moll et al. (2021); Von Moll et al. (2022b); Von Moll et al. (2022a), the turret defense and perimeter defense games, in which the defenders are restricted to the boundary of the goal region, are also analyzed via the HJI method, and the equilibrium controls are obtained by solving the ODE system (6).

Apart from the computation above, the HJI method is also used as a tool to verify the value function sufficiently. Letting $\mathcal{X}$ be a subset of the state space with $\mathcal{M}\subset \mathcal{X}$, if a function V(x) is such that

1. It is continuously differentiable everywhere over $\mathcal{X}\backslash \mathcal{M}$;

2. It satisfies the HJI Eq. 4 over $\mathcal{X}\backslash \mathcal{M}$;

3. It equals to Φ(x) on the boundary of $\mathcal{M}$,

then V(x) is the value function of the game over $\mathcal{X}$. Examples can be found in Garcia et al. (2020, 2021); Yan et al. (2022).

4 Construction of barriers and winning regions

In this section, we review the barrier construction for multiple/single player(s) against one opponent in five interesting and representative M-RA differential games by detailing the game description and barrier construction individually. We will omit the resulting winning regions which interested readers can find in the related papers, as by definition, they follow from the barriers directly.

4.1 Two-dimensional bounded convex game region

4.1.1 Game description

The game region Ω is a two-dimensional (2D) closed convex region and the splitting hypersurface $\mathcal{T}$ is a straight line with length ℓ such that Ω_play and Ω_goal are non-empty (see Figure 1, where O is the origin). The point capture is considered, i.e., r_i = 0 for all 1 ≤ i ≤ N_p. Homogeneous pursuers and evaders are considered, that is, players in each team have the same speed. The pursuers are assumed to be faster than the evaders, and the speed ratio is denoted by α > 1.

4.1.2 Barrier construction

For this game, let x = [x,y]^⊤ for any vector $\mathit{x}\in {\mathbb{R}}^2$. We focus on the barrier for the pursuit team against one evader. A pursuer is active and contributes to the barrier construction, if it dominates at least one point in the splitting line $\mathcal{T}$ against other pursuers. Since only barrier contributors are necessary for barrier computation by definition, we determine all active pursuers first. If the pursuit team has a unique active pursuer (say P_i), then the barrier $\mathcal{B}({\mathit{x}}_{P_i})$, consisting of three curves, is computed as follows: $\mathcal{B}({\mathit{x}}_{P_i})=\tilde{\mathcal{B}}({\mathit{x}}_{P_i})\cap {\mathrm{\Omega }}_{\text{play}}$ and $\tilde{\mathcal{B}}({\mathit{x}}_{P_i})={\bigcup }_{k=1}^3{\tilde{\mathcal{B}}}_k({\mathit{x}}_{P_i})$, where

$\begin{array}{cc}\hfill {\tilde{\mathcal{B}}}_1\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\alpha \Vert \mathit{x}-{\mathit{x}}_1{\Vert }_2-\Vert {\mathit{x}}_{P_i}-{\mathit{x}}_1{\Vert }_2=0,x\le {\sigma }_1,y>0\right\},\hfill \\ \hfill {\tilde{\mathcal{B}}}_2\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\left({\alpha }^2-1\right)y^2-{\left(x-x_{P_i}\right)}^2-\left(1-1/{\alpha }^2\right)y_{P_i}^2=0,x\in \left({\sigma }_1,{\sigma }_2\right),y>0\right\},\hfill \\ \hfill {\tilde{\mathcal{B}}}_3\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\alpha \Vert \mathit{x}-{\mathit{x}}_2{\Vert }_2-\Vert {\mathit{x}}_{P_i}-{\mathit{x}}_2{\Vert }_2=0,x\ge {\sigma }_2,y>0\right\},\hfill \end{array}(8)$

and ${\mathit{x}}_1={[\mathrm{0,0}]}^{\top },{\mathit{x}}_2={[\ell ,0]}^{\top },{\sigma }_1=x_{P_i}/{\alpha }^2$ and ${\sigma }_2=(1-1/{\alpha }^2)\ell +x_{P_i}/{\alpha }^2$. If the pursuit team consists of two active pursuers (say P_c = {P₁, P₂} and assume $x_{P_1}<x_{P_2}$), then the barrier $\mathcal{B}({\mathit{x}}_{P_c})$, consisting of five curves, is computed as follows: $\mathcal{B}({\mathit{x}}_{P_c})=\tilde{\mathcal{B}}({\mathit{x}}_{P_c})\cap {\mathrm{\Omega }}_{\text{play}}$ and $\tilde{\mathcal{B}}({\mathit{x}}_{P_c})={\bigcup }_{k=1}^5{\tilde{\mathcal{B}}}_k({\mathit{x}}_{P_c})$, where

$\begin{array}{cc}\hfill {\tilde{\mathcal{B}}}_1\left({\mathit{x}}_{P_c}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\alpha {‖\mathit{x}-{\mathit{x}}_1‖}_2-{‖{\mathit{x}}_{P_1}-{\mathit{x}}_1‖}_2=0,x\le {\sigma }_1,y>0\right\},\hfill \\ \hfill {\tilde{\mathcal{B}}}_2\left({\mathit{x}}_{P_c}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\left({\alpha }^2-1\right)y^2-{\left(x-x_{P_1}\right)}^2-\left(1-1/{\alpha }^2\right)y_{P_1}^2=0,x\in \left({\sigma }_1,{\sigma }_2\right),y>0\right\},\hfill \\ \hfill {\tilde{\mathcal{B}}}_3\left({\mathit{x}}_{P_c}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\alpha {‖\mathit{x}-{\mathit{x}}_2‖}_2-{‖{\mathit{x}}_{P_2}-{\mathit{x}}_2‖}_2=0,x\in \left[{\sigma }_2,{\sigma }_3\right],y>0\right\},\hfill \\ \hfill {\tilde{\mathcal{B}}}_4\left({\mathit{x}}_{P_c}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\left({\alpha }^2-1\right)y^2-{\left(x-x_{P_2}\right)}^2-\left(1-1/{\alpha }^2\right)y_{P_2}^2=0,x\in \left({\sigma }_3,{\sigma }_4\right),y>0\right\},\hfill \\ \hfill {\tilde{\mathcal{B}}}_5\left({\mathit{x}}_{P_c}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|\alpha {‖\mathit{x}-{\mathit{x}}_3‖}_2-{‖{\mathit{x}}_{P_2}-{\mathit{x}}_3‖}_2=0,x\ge {\sigma }_4,y>0\right\},\hfill \end{array}(9)$

where ${\mathit{x}}_1={[\mathrm{0,0}]}^{\top },{\mathit{x}}_2={[x_2,0]}^{\top },{\mathit{x}}_3={[\ell ,0]}^{\top },{\sigma }_1=x_{P_1}/{\alpha }^2,{\sigma }_2=(1-1/{\alpha }^2)x_2+x_{P_1}/{\alpha }^2,{\sigma }_3=(1-1/{\alpha }^2)x_2+x_{P_2}/{\alpha }^2,{\sigma }_4=(1-1/{\alpha }^2)\ell +x_{P_2}/{\alpha }^2$and $x_2=({‖{\mathit{x}}_{P_2}‖}_2^2-{‖{\mathit{x}}_{P_1}‖}_2^2)/(2(x_{P_2}-x_{P_1}))$. The barrier $\tilde{\mathcal{B}}({\mathit{x}}_{P_c})$ without considering the boundary of the play region is shown in Figure 3A, and the complete barrier $\mathcal{B}({\mathit{x}}_{P_i})$ in Figure 3B. More generally, if the pursuit team has more than two active pursuers, it has been proved that any point on the underlying barrier can be determined by at most two active pursuers. With this observation, the barrier is constructed by concatenating the two-pursuer barriers for all pairs of adjacent active pursuers along $\mathcal{T}$. We refer interested readers to Yan et al. (2020) for more details.

FIGURE 3

FIGURE 3. Barrier construction for two-dimensional (2D) bounded convex game region. (A) No boundary for play region; (B) Bounded play region.

4.2 Three-dimensional game region

4.2.1 Game description

The game region Ω is the whole three-dimensional (3D) space and $\mathcal{T}$ is a plane such that Ω_play and Ω_goal are two-half spaces. The point capture and radius capture are both considered, that is, r_i ≥ 0 for 1 ≤ i ≤ N_p. Pursuers and evaders are heterogeneous in the sense that players in each team may have different speeds and pursuers may have different capture radii. The pursuers are assumed to be faster than the evaders, that is, $v_{P_i}>v_{E_j}$ for all 1 ≤ i ≤ N_p and 1 ≤ j ≤ N_e.

4.2.2 Barrier construction

We focus on the barrier for the pursuit team against one evader. The capture strategy in Yan et al. (2022) indicates that the barrier is equivalent to the case where the dominance region of the evader which is proved to be strictly convex before the capture occurs, is tangent to the goal region. Formally, the barrier can be computed as follows

$\mathcal{B}\left({\mathit{x}}_{P_c}\right)=\left\{{\mathit{x}}_{E_j}\in \mathrm{\Omega }\mid \left\{\mathit{x}\in \mathrm{\Omega }\mid \Vert \mathit{x}-{\mathit{x}}_{P_i}{\Vert }_2-{\alpha }_{ij}\Vert \mathit{x}-{\mathit{x}}_{E_j}{\Vert }_2-r_i\ge 0,\forall 1\le i\le N_p\right\}\mathbf{\text{tangent to}}{\mathrm{\Omega }}_{\text{goal}}\right\}.$

Since Yan et al. (2022) proves that in E_j’s dominance region, the unique point closest to the goal region can be determined by at most three pursuers, checking the tangent property for all pursuer combinations with no more than three pursuers would be sufficient to cover all points of the barrier, improving the computational efficiency drastically. The extension to a convex play region with an exit is also discussed in Yan et al. (2022).

4.3 Limited evasion lifetime

4.3.1 Game description

The game region Ω is the whole 2D plane and $\mathcal{T}$ is a straight line separating Ω into two disjoint half planes Ω_play and Ω_goal. The radius capture is considered, and the pursuer is faster than the evader. Apart from the above, the evader has to reach the goal region Ω_goal within a limited lifetime t_a (t_a > 0) prior to the capture or the evader loses the game otherwise.

4.3.2 Barrier construction

We focus on the barrier for one pursuer against one evader. First, we compute the barrier for the game without lifetime limitation which directly follows from Section 4.1, as indicated by ${\mathcal{B}}^{\infty }$ in Figure 4. Then, the points at the barrier which correspond to the capture/reach time larger than t_a (the dashed part of ${\mathcal{B}}^{\infty }$), are discarded. The barrier is further completed considering the following two cases. The first one is that, the lifetime is the only active constraint and thus the optimal evasion strategy is moving directly towards Ω_goal and reaching Ω_goal exactly when the lifetime runs out, as depicted in green. The second one is that, both the lifetime and the capture both are active constraints, and the evader reaches the goal region exactly when the capture happens and the lifetime is up at the same time, as depicted in magenta in Figure 4. Following this, the barrier is computed as follows: if $|y_{P_i}|>v_{P_i}{t}_a+r_i$,

$\mathcal{B}\left({\mathit{x}}_{P_i}\right)=\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|x\in \mathbb{R},y=v_{P_i}{t}_a/{\alpha }_{ij}\right\},(10)$

and $\mathcal{B}({\mathit{x}}_{P_i})={\bigcup }_{k=1}^5{\mathcal{B}}_k({\mathit{x}}_{P_i})$ otherwise, where

$\begin{array}{cc}\hfill {\mathcal{B}}_1\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|y=v_{P_i}{t}_a/{\alpha }_{ij},x\le x_1\right\},\hfill \\ \hfill {\mathcal{B}}_2\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|{\left(x-x_1\right)}^2+y^2=v_{P_i}^2{t}_a^2/{\alpha }_{ij}^2,x_1<x<x_2,y>0\right\},\hfill \\ \hfill {\mathcal{B}}_3\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|x=x^{*}-d_1{d}_2/{\alpha }_{ij}^2,y=d_1\sqrt{{\alpha }_{ij}^2-d_2^2}/{\alpha }_{ij}^2\right\},\hfill \\ \hfill {\mathcal{B}}_4\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|{\left(x-x_3\right)}^2+y^2=v_{P_i}^2{t}_a^2/{\alpha }_{ij}^2,x_4<x<x_5,y>0\right\},\hfill \\ \hfill {\mathcal{B}}_5\left({\mathit{x}}_{P_i}\right)& =\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|y=v_{P_i}{t}_a/{\alpha }_{ij},x\ge x_5\right\}.\hfill \end{array}(11)$

The variable x* in (11) is as follows: $x^{*}\in \mathbb{R}$ if $|y_{P_i}|\ge r_i$ and $x^{*}\in \left\{\left.x\in \mathbb{R}\right||x-x_{P_i}|\ge \sqrt{r_i^2-y_{P_i}^2}\right\}$ otherwise, and

$\left\{\begin{array}{c}{x}_1=x_{P_i}-\sqrt{{\left(v_{P_i}{t}_a+r_i\right)}^2-y_{P_i}^2}\hfill \\ x_2=x_1+\frac{v_{P_i}{t}_a\sqrt{{\left(v_{P_i}{t}_a+r_i\right)}^2-y_{P_i}^2}}{{\alpha }_{ij}^2\left(v_{P_i}{t}_a+r_i\right)}\hfill \end{array}\right.\left\{\begin{array}{c}{x}_5=x_{P_i}+\sqrt{{\left(v_{P_i}{t}_a+r_i\right)}^2-y_{P_i}^2}\hfill \\ x_4=x_5-\frac{v_{P_i}{t}_a\sqrt{{\left(v_{P_i}{t}_a+r_i\right)}^2-y_{P_i}^2}}{{\alpha }_{ij}^2\left(v_{P_i}{t}_a+r_i\right)}.\hfill \end{array}\right.(12)$

The complete barrier $\mathcal{B}({\mathit{x}}_{P_i})$ is the union of these colored solid lines. We refer interested readers to Yan et al. (2021b) for details.

FIGURE 4

FIGURE 4. Barrier construction for 2D reach-avoid differential games with limited evasion lifetime (Yan et al., 2021b).

4.4 View of the evasion team

4.4.1 Game description

The game region Ω is the 2D plane and $\mathcal{T}$ is a straight line. The pursuit team $\mathcal{P}$ has a unique pursuer, say P, and the evasion team $\mathcal{E}$ consists of two evaders E₁ and E₂. We focus on the point capture. The pursuer is faster than the evaders, i.e., ${\alpha }_j=v_P/v_{E_j}>1$ for j = 1, 2. We demonstrate the existence of cooperative strategies among evaders in this example. The evaders are assumed to know which evader is currently being chased by the pursuer.

4.4.2 Barrier construction

The barrier in this case splits the state space into three disjoint parts: Under the players’ optimal strategies, the first one corresponds to no captured evader, the second corresponds to one captured evader and the third corresponds to two captured evaders. Since it takes the pursuer some time to capture the first-pursued evader (if the capture is guaranteed) before pursuing the second-pursued evader, the Apollonius circle is generalised to tackle the scenario where the pursuer starts to pursue the evader when the latter has already moved for a time interval δ. Formally, the dominance region accounting for a time difference between the pursuit and evasion of P and E_j, called δ-Apollonius circle, is defined as follows

${\mathbb{E}}_j^{\delta }=\left\{\mathit{x}\in {\mathbb{R}}^2\mid {\alpha }_j\Vert \mathit{x}-{\mathit{x}}_{E_j}{\Vert }_2=\Vert \mathit{x}-{\mathit{x}}_P{\Vert }_2+v_P\delta \right\}.(13)$

If E_j moves freely before P pursues it for a time period δ, then based on the δ-Apollonius circle, the barrier is computed as follows

${\mathcal{B}}_j^{\delta }\left({\mathit{x}}_P\right)=\left\{\left.\mathit{x}\in {\mathbb{R}}^2\right|x=x^{*}-ab,y=a\sqrt{1-b^2},x^{*}\in \mathcal{P}\right\},(14)$

where ${\mathit{x}}^{*}={[x^{*},0]}^{\top }$, $a={\alpha }_j\Vert {\mathit{x}}^{*}-{\mathit{x}}_P{\Vert }_2+v_{E_j}\delta$ and b = α_j(x* − x_P)/‖x* − x_P‖₂. The feasible set $\mathcal{P}$ for x* is determined as follows. If $\delta \le \frac{(1-{\alpha }_j^2)|y_P|}{{\alpha }_j{v}_{E_j}}$, then $\mathcal{P}=\mathbb{R}$, and the barrier is illustrated in Figure 5A. If $\delta >\frac{(1-{\alpha }_j^2)|y_P|}{{\alpha }_j{v}_{E_j}}$, then $\mathcal{P}$ is given by

$\mathcal{P}=\left\{\left.x\in \mathbb{R}\right||x-x_P|\ge \sqrt{{\left(\frac{{\alpha }_j{v}_{E_j}\delta }{1-{\alpha }_j^2}\right)}^2-y_P^2}\right\},(15)$

and the barrier is illustrated in Figure 5B. Then, the barrier for two evaders against one pursuer follows by combining the common one-versus-one barrier without time difference and the proposed one-versus-one barrier with a time difference, where an aiding strategy between two evaders may occur. More specifically, if P pursues E₁ first and then E₂, the aiding strategy describes that E₁ moves away from the goal region to aid E₂’s evasion, such that E₂ reaches the best relative position to escape when E₁ is captured. This strategy implies that one evader may need to sacrifice itself to save the other evader, which is frequently observed between prey animals. As an illustration, Figure 5C indicates that if P pursues E₁ first, then the game space is divided by the orange curve into two disjoint regions ${\mathcal{Q}}_P$ and ${\mathcal{Q}}_E$ such that if E₂ lies in ${\mathcal{Q}}_P$ currently, then P can ensure the capture of E₂ after capturing E₁, while if E₂ lies in ${\mathcal{Q}}_E$, E₂ is able to reach Ω_goal without being captured. Figure 5D shows the case when P pursues E₂ first. Combining these two cases, we conclude that the pursuer should pursue E₁ first. We refer interested readers to Yan et al. (2021a) for details.

FIGURE 5

FIGURE 5. Barrier construction for 2D reach-avoid differential games with one pursuer and two evaders (Yan et al., 2021a). (A) Small time difference Δ; (B) Large time difference Δ; (C) Winning spaces when P pursues E₁ first; (D) Winning spaces when P pursues E₂ first.

4.5 The lady in the lake with multiple pursuers

4.5.1 Game description

We extend the classical game the Lady in the Lake (Isaacs, 1965) to multiple pursuers. The game region Ω is the whole two-dimensional plane and $\mathcal{T}$ is a circle such that Ω_play is the disk inside $\mathcal{T}$ and Ω_goal is the remainder. The evasion team has a unique evader, i.e., the lady. The point capture is considered, and the pursuers are assumed to be faster than the evader. The pursuers are restricted to the circle $\mathcal{T}$ and maintain a uniform distribution along $\mathcal{T}$ by cooperation, as shown in Figure 6A.

FIGURE 6

FIGURE 6. The Lady in the Lake with multiple pursuers. Game description (A); the barrier does not exist (B) and the barrier occurs (C) for one pursuer and one evader.

4.5.2 Barrier construction

Since the pursuers are uniformly distributed, the goal of the evader is to penetrate $\mathcal{T}$ through a point between two adjacent pursuers. Yan et al. (2017) reveals that if the speed radio is less than a constant which only depends on the number of pursuers, then there is no barrier and the evader can always escape. The escape strategies are classified into two types, depending on their relative positions. Roughly speaking, the evader escapes directly along a straight line if its distance to the closest pursuer is long enough for a successful escape, and otherwise, the evader needs to go back to the center, then adjust its relative position to the pursuers to create a better escape condition and finally escapes directly along a straight line. If the speed ratio is greater than or equal to this constant, then the barrier emerges. In summary, the barrier computation is as follows. Let α₀ ∈ (1, + ∞) be the unique solution of the equation

$\pi /N_p+\arccos \left(1/{\alpha }_0\right)-\sqrt{{\alpha }_0^2-1}=0.(16)$

If α < α₀, then $\mathcal{B}({\mathit{x}}_{P_c})=\varnothing$. If α ≥ α₀, then $\mathcal{B}({\mathit{x}}_{P_c})={\bigcup }_{i=1}^{N_p}{\mathcal{B}}_i({\mathit{x}}_{P_c})$, where

$\begin{array}{c}\hfill {\mathcal{B}}_i\left({\mathit{x}}_{P_c}\right)=\left\{\left.\left(\rho ,{\theta }_i\right)\right||{\theta }_i|=\arccos \left(\frac{R}{\alpha \rho }\right)-\frac{\sqrt{{\alpha }^2{\rho }^2-R^2}}{R}-\arccos \left(\frac{1}{\alpha }\right)+\sqrt{{\alpha }^2-1},\rho \in \left[{\rho }_0,R\right],|{\theta }_i|\le \pi /N_p\right\},\end{array}(17)$

and ρ₀ is the solution to the equation in (17) for θ_i = π/N_p. We depict one pursuer case for an illustration. In Figure 6B, α < α₀ holds and the red curve splits the game space into ϒ₂ and ϒ₃, such that E has different strategies separately as stated above, where ϒ₁ is the circle that E should enter if it lies in ϒ₂. In Figure 6C, α ≥ α₀ holds and the (orange) barrier emerges. We refer interested readers to Yan et al. (2017) for details.

5 Task allocation

Task allocation, a popular task planning strategy, focuses on assigning groups of simple tasks to individual players for execution. When applied to M-RA differential games, the player configurations, availabilities and capabilities need to be considered (Smith et al., 2009; Bajaj and Bopardikar, 2019; Yan et al., 2020; Yan et al., 2022; Bajaj et al., 2021; Antonyshyn et al., 2022; Velhal et al., 2022). In this section, we first introduce an integer linear programming formulation for capturing the most number of evaders in Section 5.1 and then propose a polynomial approximation algorithm in Section 5.2.

5.1 Integer linear programming

From the pursuit team’s perspective, the goal is, for each evader, to designate a pursuit coalition which is capable of capturing the evader before it enters the goal region. If the barrier of the game is constructed, a pursuit coalition is adequate if the evader and the pursuit coalition lie in the PWR. In this way, we collect the outcomes of all pursuit coalition and evader pairs prior to the game execution. Then, we match pursuit coalitions with the evaders such that the most number of evaders are captured. This task allocation problem can be formulated as a 0–1 integer linear program as follows.

Suppose that the size of the pursuit coalition is less or equal to N_c (N_c ≤ N_p). Then the pursuit team $\mathcal{P}$ consists of $N_{\mathit{all}}=C_{N_p}^1+C_{N_p}^2+\cdots +C_{N_p}^{N_c}$ possible coalitions: $C_{N_p}^1$ one-pursuer coalitions, $C_{N_p}^2$ two-pursuer coalitions, and so on. Let $\mathcal{G}=({\mathcal{V}}_P\cup {\mathcal{V}}_E,\mathcal{E})$ be an undirected bipartite graph consisting of two independent vertex sets ${\mathcal{V}}_P$, ${\mathcal{V}}_E$ and a set of edges $\mathcal{E}$. The vertex set ${\mathcal{V}}_P$ consists of all N_all pursuit coalitions, and ${\mathcal{V}}_E$ represents the set of evaders. The edge connecting vertex $P_c\in {\mathcal{V}}_P$ and vertex $E_j\in {\mathcal{V}}_E$ is denoted by e_cj. An edge $e_{cj}\in \mathcal{E}$ if and only if P_c is capable of capturing E_j before the latter enters Ω_goal, while any strict subcoalition of P_c cannot. The goal of the task allocation here is to find a matching in $\mathcal{G}$ that contains a maximum number of evaders. Since a pursuer can only appear in at most one pursuit coalition for an executable matching, a conflict graph $\mathcal{C}=(\mathcal{E},\bar{\mathcal{E}})$ is introduced to account for such conflicts among the pursuit coalitions. Each vertex in $\mathcal{C}$ corresponds to an edge $e\in \mathcal{E}$ of $\mathcal{G}$, and an edge $\bar{e}\in \bar{\mathcal{E}}$ if and only if the vertexes connected by $\bar{e}$, say $e_{cj},e_{pq}\in \mathcal{E}$, have no shared pursuers, i.e., P_c ∩ P_p is empty. Formally, the task allocation problem is to find a matching that solves the following integer linear program

$\begin{array}{cccc}\hfill & \text{maximize}\hfill & \hfill & {\sum }_{e_{cj}\in \mathcal{E}}{x}_{cj}\hfill \\ \hfill & \text{subject\hspace{0.17em}to}\hfill & \hfill & \sum _{P_c\in {\mathcal{V}}_P}{x}_{cj}\le 1\forall E_j\in {\mathcal{V}}_E,\sum _{E_j\in {\mathcal{V}}_E}{x}_{cj}\le 1\forall P_c\in {\mathcal{V}}_P,\hfill \\ \hfill & \hfill & \hfill & x_{cj}+x_{pq}\le 1\forall \left(e_{cj},e_{pq}\right)\in \bar{\mathcal{E}},\hfill \\ \hfill & \hfill & \hfill & x_{cj}\in \left\{\mathrm{0,1}\right\}\forall e_{cj}\in \mathcal{E},x_{cj}=0\forall e_{cj}\notin \mathcal{E},\hfill \end{array}(18)$

where x_cj = 1 indicates the assignment of pursuit coalition P_c to capture E_j, and x_cj = 0 means no assignment.

5.2 Polynomial approximation algorithm

Since problem (18) is a special constrained matching problem (Tanimoto et al., 1978) and proved to be NP-hard (Yan et al., 2022), solving (18) is intractable when the number of players is large. Fortunately, Yan et al. (2022) shows that there exist constant-factor polynomial algorithms for problem (18), and further proposes a 1/N_c-approximation polynomial algorithm called Sequential Matching algorithm. In this algorithm, First, polynomial algorithms (e.g., maximum network flow) are used to compute the maximum matching of the subgraph of $\mathcal{G}$ which only considers the pursuit vertexes containing one pursuer. Then, the matched players are removed from $\mathcal{G}$, and we compute another maximum matching of the subgraph of the new $\mathcal{G}$ which only considers the pursuit vertexes containing two pursuers. Repeat the process until $\mathcal{G}$ has no vertexes at either side, or pursuit coalitions with N_c pursuers have all been considered. Finally, a 1/N_c-factor approximation matching solution is obtained by merging all these maximum matchings which have no shared vertexes by construction.

6 Cooperative strategies

Based on the results of the game of kind, the game of degree needs to provide the strategies for the players to ensure their winnings and optimize some metrics at the same time. In this section, we review three types of dominance region based cooperative strategies, with a focus on the pursuers.

6.1 Voronoi-based strategy

Voronoi partitions are widely used for generating cooperative strategies for the players, usually when they all have the same speed. There are three popular Voronoi-based pursuit strategies: area-based, point-based, and relay strategies. The area-based pursuit strategy is aiming at minimizing the area of the evader’s Voronoi cell (Pan et al., 2012). The point-based pursuit strategy requires that each pursuer moves towards a specific point in the evader’s dominance region, such as the farthest point from the evader’s current position, and the point closest to the goal region (Yan et al., 2019b; Garcia et al., 2020; Yan et al., 2022). The relay pursuit strategy allows the pursuers to pursue the evader in a relay way based on whether the evader is in its dominance region against the other pursuers.

6.2 Apollonius-circle based strategy

As for unequal speed scenarios, the Apollonius circle is used to design cooperative strategies for the pursuers. Most of Apollonius-based pursuit strategies are point-based. For instance, since the evader’s dominance region, formed by the intersection of all one-to-one Apollonius circles, is strictly convex, the point on the dominance region closest to a convex goal region (if they are disjoint) is unique and thus moving towards this point under feedback strategies can ensure the pursuit winning (Yan et al., 2019b). However, the singularity needs to be resolved when the non-convex goal regions are considered Von Moll et al. (2020).

6.3 Convex optimization based strategy

It is difficult to use Voronoi-based or Apollonius-based strategies when the pursuers have positive capture radii, due to the lack of the closed-form representation of the dominance region. Inspired by the function-based dominance region, Yan et al. (2022) proposed a convex optimization based pursuit strategy which applies to both point capture and radius capture cases. For multiple pursuers against one evader, if the evader’s dominance region is disjoint from the goal region, then the point x_I (may be non-unique) in the dominance region closest to the goal region is computed by solving the convex optimization problem

$\begin{array}{cccc}\hfill & \underset{\left(\mathit{x},\mathit{y}\right)\in {\mathbb{R}}^n\times {\mathbb{R}}^n}{\text{minimize}}\hfill & \hfill & \Vert \mathit{x}-\mathit{y}{\Vert }_2\hfill \\ \hfill & \text{\hspace{0.17em}subject\hspace{0.17em}to}\hfill & \hfill & f_{ij}\left(\mathit{x}\right)\ge 0,g\left(\mathit{y}\right)\le 0,\forall i\in c,\hfill \end{array}(19)$

where f_ij is defined in Definition 3 and $g:{\mathbb{R}}^n\to \mathbb{R}$ is such that ${\mathrm{\Omega }}_{\text{goal}}=\{\mathit{x}\in {\mathbb{R}}^n\mid g(\mathit{x})\le 0\}$ is a non-empty, closed convex region. The convexity of the problem follows from the fact that the evader’s dominance region and the goal region are both convex. Then, the current control input of each pursuer is defined as the direction pointing to x_I, i.e., the pursuer moves towards x_I. Yan et al. (2022) shows that if the pursuers are faster, then x_I is unique and this feedback strategy is able to guarantee that the dominance region never approaches the goal region, leading to a guaranteed pursuit winning.

7 Discussion

Being a relatively new field of study, many research questions remain open for M-RA differential games. In this section, we discuss the limitations in the existing literature and point out directions for future developments, from the following aspects of the games inspired by Shishika and Kumar (2020): player dynamics, sequential capture, spatial-temporal coupling, fast evaders and partial information.

7.1 Player dynamics

We assumed that each player is modelled by simple motion and thus can change its heading instantaneously. As discussed above, this dynamical model is a suitable abstraction for mobile robots or robotic vehicles which have limited speed and high maneuverability. However, such abstractions may generate strategies which fail to complete the tasks, since some constraints ignored in the abstraction have a crucial effect on the strategy synthesis. Examples of the constraints include minimum turning radius, maximum acceleration, and external forces. Taking these dynamical constraints into account will inevitably complicate the strategy synthesis.

7.2 Sequential capture

If the pursuer is allowed to capture multiple evaders sequentially, then this scenario will involve a dynamic vehicle routing problem (Bopardikar et al., 2010) in an adversarial setting. This cannot be handled with existing barrier construction methods which only focus on myopic capture, i.e., the capture of the evader being pursued without reasoning the pursuit after the capture. Taking sequential capture into consideration when synthesizing strategies will lead to many interesting strategic behaviors, and constructive results have been presented when the evaders are assumed to arrive in a probabilistic spatio-temporal manner (Smith et al., 2009; Bajaj and Bopardikar, 2019; Bajaj et al., 2021). For example, some of the evaders may lure the pursuers away from the goal region so that other evaders can reach the goal region. When constructing the barriers for capturing multiple evaders, the pursuers may chase the evaders that are further away first and the close ones afterwards.

7.3 Spatial-temporal coupling

The task allocation method in Section 5 assumes that each pursuit coalition plays a game against an evader independently. However, since all players operate in a shared environment simultaneously, the players’ trajectories in different games are coupled spatially and temporally. Such coupling may lead to future collision and can also be leveraged to design wiser strategies. Taking the coupling of the future paths between different matching pairs into account is worth studying.

7.4 Fast evaders

Most of existing results are provided when the pursuers are faster or equal to the evaders. The most significant consequence of this constraint is that the evader’s dominance region, represented by either Voronoi cell, Apollonius circle or non-negative level set of a function, is convex. This convexity property ensures that the evader dominates all points along the straight line from its current position to any goal point in its dominance region, implying a capture-free path regardless of the pursuers’ strategies. However, the game with faster evaders is fundamentally different, because the capture requires more complicated cooperation among the pursuers to offset the speed disadvantages, or leverage the characteristics of the game region (e.g, boundaries and convexity).

7.5 Partial information

The assumption in the existing works that each player has full knowledge of the positions and speeds of all other players, may be invalid in many realistic situations due to the adversarial objectives. First, the pursuers have a limited detection range out of which the information about evaders and the environment may be unavailable. Second, even if the evaders are detected, measurement errors exist and vary depending on the sensing devices. Third, if the number of evaders within the detection range is large, then counting or locating all possible evaders in a dense swarm raises a big challenge to the detection capabilities of the pursuers.

8 Conclusion

In this work, we reviewed the recent progress in M-RA differential games. We provided background on game elements, application and problems of interest. We introduced two common methods, geometric method and HJI method, for solving M-RA differential games. We presented a review of barrier construction (winning regions follow immediately) for multiple players against one opponent player in several games. We presented an integer linear programming formulation and its approximation algorithm to tackle multiple versus multiple cases using the results of multiple versus one and the maximum matching. We presented three dominance region based pursuit strategies, depending on the speed ratio and the capture radius. Finally, we discussed several limitations in the current problem formulation and identified the corresponding trends for future research.

Author contributions

RY contributed to conception, design of the study, data collection and analysis. RY wrote the first draft of the manuscript, and RD wrote Sections 3.2, 6 of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version. The work of RY was completed at Tsinghua University.

Funding

The work of RY, RD, ZS, and YZ was supported by the Science and Technology Innovation 2030-Key Project of “New Generation Artificial Intelligence” under Grant 2020AAA0108200. The work of XD was sponsored by Shanghai Pujiang Program under Grant 22PJ1404900.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Akilan, Z., and Fuchs, Z. (2017). “Zero-sum turret defense differential game with singular surfaces,” in 2017 IEEE Conference on Control Technology and Applications (CCTA), Maui, HI, 27–30 August, 2017, 2041–2048.

CrossRef Full Text | Google Scholar

Antonyshyn, L., Silveira, J., Givigi, S., and Marshall, J. (2022). Multiple mobile robot task and motion planning: A survey. ACM Computing Surveys (CSUR). doi:10.1145/3564696

CrossRef Full Text | Google Scholar

Bajaj, S., and Bopardikar, S. D. (2019). “Dynamic boundary guarding against radially incoming targets,” in 2019 IEEE 58th Conference on Decision and Control (CDC), Nice, France, 11–13 December, 2019 (IEEE), 4804–4809.

CrossRef Full Text | Google Scholar

Bajaj, S., Torng, E., and Bopardikar, S. D. (2021). “Competitive perimeter defense on a line,” in 2021 American Control Conference (ACC), New Orleans, LA, 25–28 May, 2021 (IEEE), 3196–3201.

CrossRef Full Text | Google Scholar

Blaquière, A., Gérard, F., and Leitmann, G. (1969). Quantitative and qualitative games. New York: Academic Press.

Google Scholar

Bopardikar, S. D., Smith, S. L., Bullo, F., and Hespanha, J. P. (2010). Dynamic vehicle routing for translating demands: Stability analysis and receding-horizon policies. IEEE Trans. Autom. Control 55, 2554–2569. doi:10.1109/tac.2010.2049278

CrossRef Full Text | Google Scholar

Chen, X., and Yu, J. (2022). Reach-avoid games with two heterogeneous defenders and one attacker. IET Control Theory Appl. 16, 301–317. doi:10.1049/cth2.12226

CrossRef Full Text | Google Scholar

Chen, M., Zhou, Z., and Tomlin, C. J. (2016). Multiplayer reach-avoid games via pairwise outcomes. IEEE Trans. Autom. Control 62, 1451–1457. doi:10.1109/tac.2016.2577619

CrossRef Full Text | Google Scholar

Chen, M., Herbert, S. L., Vashishtha, M. S., Bansal, S., and Tomlin, C. J. (2018). Decomposition of reachable sets and tubes for a class of nonlinear systems. IEEE Trans. Autom. Control 63, 3675–3688. doi:10.1109/tac.2018.2797194

CrossRef Full Text | Google Scholar

Deng, R., Yan, R., Zhang, W., Shi, Z., and Zhong, Y. (2021). “Receding horizon defense strategy for reach-avoid games with uncertainties via pairwise outcomes,” in 2021 40th Chinese Control Conference (CCC), Shanghai, China, 26–28 July, 2021 (IEEE), 5401–5406.

CrossRef Full Text | Google Scholar

Dubins, L. E. (1957). On curves of minimal length with a constraint on average curvature, and with prescribed initial and terminal positions and tangents. Am. J. Math. 79, 497–516. doi:10.2307/2372560

CrossRef Full Text | Google Scholar

Fisac, J. F., Chen, M., Tomlin, C. J., and Sastry, S. S. (2015). “Reach-avoid problems with time-varying dynamics, targets and constraints,” in Proceedings of the 18th international conference on hybrid systems: computation and control, Seattle, Washington, April, 2015, 11–20.

CrossRef Full Text | Google Scholar

Fu, H., and Liu, H. H.-T. (2020). Guarding a territory against an intelligent intruder: Strategy design and experimental verification. IEEE/ASME Trans. Mechatronics 25, 1765–1772. doi:10.1109/tmech.2020.2996901

CrossRef Full Text | Google Scholar

Fu, H., and Liu, H. H.-T. (2021). An isochron-based solution to the target defense game against a faster invader. IEEE Control Syst. Lett. 6, 1352–1357. doi:10.1109/lcsys.2021.3092950

CrossRef Full Text | Google Scholar

Garcia, E., Casbeer, D. W., Fuchs, Z. E., and Pachter, M. (2018). Cooperative missile guidance for active defense of air vehicles. IEEE Trans. Aerosp. Electron. Syst. 54, 706–721. doi:10.1109/taes.2017.2764269

CrossRef Full Text | Google Scholar

Garcia, E., Casbeer, D. W., and Pachter, M. (2019a). Design and analysis of state-feedback optimal strategies for the differential game of active defense. IEEE Trans. Autom. Control 64, 553–568.

Google Scholar

Garcia, E., Casbeer, D. W., Von Moll, A., and Pachter, M. (2019b). “Cooperative two-pursuer one-evader blocking differential game,” in 2019 American Control Conference (ACC), Philadelphia, PA, 10–12 July, 2019 (IEEE), 2702–2709.

CrossRef Full Text | Google Scholar

Garcia, E., Casbeer, D. W., and Pachter, M. (2020). Optimal strategies for a class of multi-player reach-avoid differential games in 3D space. IEEE Robotics Autom. Lett. 5, 4257–4264. doi:10.1109/lra.2020.2994023

CrossRef Full Text | Google Scholar

Garcia, E., Casbeer, D. W., Von Moll, A., and Pachter, M. (2021). Multiple pursuer multiple evader differential games. IEEE Trans. Autom. Control 66, 2345–2350. doi:10.1109/tac.2020.3003840

CrossRef Full Text | Google Scholar

Getz, W. M., and Pachter, M. (1981). Two-target pursuit-evasion differential games in the plane. J. Optim. Theory Appl. 34, 383–403. doi:10.1007/bf00934679

CrossRef Full Text | Google Scholar

Guerrero-Bonilla, L., Egerstedt, M., and Dimarogonas, D. V. (2021). “Area defense and surveillance on rectangular regions using control barrier functions,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, 27 September–01 October, 2021 (IEEE), 8166–8172.

CrossRef Full Text | Google Scholar

Huang, H., Ding, J., Zhang, W., and Tomlin, C. J. (2014). Automation-assisted capture-the-flag: A differential game approach. IEEE Trans. Control Syst. Technol. 23, 1014–1028. doi:10.1109/tcst.2014.2360502

CrossRef Full Text | Google Scholar

Ibragimov, G., Ferrara, M., Kuchkarov, A., and Pansera, B. A. (2018). Simple motion evasion differential game of many pursuers and evaders with integral constraints. Dyn. Games Appl. 8, 352–378. doi:10.1007/s13235-017-0226-6

CrossRef Full Text | Google Scholar

Isaacs, R. (1965). Differential games. New York: Wiley.

Google Scholar

Lee, Y., and Bakolas, E. (2021). “Optimal strategies for guarding a compact and convex target set: A differential game approach,” in 2021 60th IEEE Conference on Decision and Control (CDC), Austin, TX, 14–17 December, 2021 (IEEE), 4320–4325.

CrossRef Full Text | Google Scholar

Liang, L., Deng, F., Peng, Z., Li, X., and Zha, W. (2019). A differential game for cooperative target defense. Automatica 102, 58–71. doi:10.1016/j.automatica.2018.12.034

CrossRef Full Text | Google Scholar

Liang, L., Deng, F., Lu, M., and Chen, J. (2021). Analysis of role switch for cooperative target defense differential game. IEEE Trans. Automatic Control 66, 902–909. doi:10.1109/tac.2020.2987701

CrossRef Full Text | Google Scholar

Liang, L., Deng, F., Wang, J., Lu, M., and Chen, J. (2022). A reconnaissance penetration game with territorial-constrained defender. IEEE Trans. Automatic Control 67, 6295–6302. doi:10.1109/tac.2022.3183034

CrossRef Full Text | Google Scholar

Margellos, K., and Lygeros, J. (2011). Hamilton–Jacobi formulation for reach–avoid differential games. IEEE Trans. Automatic Control 56, 1849–1861. doi:10.1109/tac.2011.2105730

CrossRef Full Text | Google Scholar

Mitchell, I. M., Bayen, A. M., and Tomlin, C. J. (2005). A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games. IEEE Trans. Automatic Control 50, 947–957. doi:10.1109/tac.2005.851439

CrossRef Full Text | Google Scholar

Mohanan, J., Manikandasriram, S., Venkatesan, R. H., and Bhikkaji, B. (2018). Toward real-time autonomous target area protection: Theory and implementation. IEEE Trans. Control Syst. Technol. 27, 1293–1300. doi:10.1109/tcst.2018.2805295

CrossRef Full Text | Google Scholar

Olsder, G. J., and Breakwell, J. V. (1974). Role determination in an aerial dogfight. Int. J. Game Theory 3, 47–66. doi:10.1007/bf01766218

CrossRef Full Text | Google Scholar

Oyler, D. W., Kabamba, P. T., and Girard, A. R. (2016). Pursuit–evasion games in the presence of obstacles. Automatica 65, 1–11. doi:10.1016/j.automatica.2015.11.018

CrossRef Full Text | Google Scholar

Pachter, M., and Getz, W. M. (1980). The geometry of the barrier in the game of two cars. Optim. Control Appl. Methods 1, 103–118. doi:10.1002/oca.4660010202

CrossRef Full Text | Google Scholar

Pachter, M., Garcia, E., and Casbeer, D. W. (2019). Toward a solution of the active target defense differential game. Dyn. Games Appl. 9, 165–216. doi:10.1007/s13235-018-0250-1

CrossRef Full Text | Google Scholar

Pan, S., Huang, H., Ding, J., Zhang, W., vić, D. M. S., and Tomlin, C. J. (2012). “Pursuit, evasion and defense in the plane,” in 2012 American Control Conference (ACC), Montreal, QC, 27–29 June, 2012 (IEEE), 4167–4173.

CrossRef Full Text | Google Scholar

Reeds, J., and Shepp, L. (1990). Optimal paths for a car that goes both forwards and backwards. Pac. J. Math. 145, 367–393. doi:10.2140/pjm.1990.145.367

CrossRef Full Text | Google Scholar

Selvakumar, J., and Bakolas, E. (2019). Feedback strategies for a reach-avoid game with a single evader and multiple pursuers. IEEE Trans. Cybern. 51, 696–707. doi:10.1109/tcyb.2019.2914869

CrossRef Full Text | Google Scholar

Shishika, D., and Kumar, V. (2018). “Local-game decomposition for multiplayer perimeter-defense problem,” in 2018 IEEE Conference on Decision and Control (CDC), Miami, FL, 17–19 December, 2018 (IEEE), 2093–2100.

CrossRef Full Text | Google Scholar

Shishika, D., and Kumar, V. (2020). “A review of multi agent perimeter defense games,” in International Conference on Decision and Game Theory for Security, College Park, MD, 28–30 October, 2020 (Springer), 472–485.

CrossRef Full Text | Google Scholar

Shishika, D., Paulos, J., and Kumar, V. (2020). Cooperative team strategies for multi-player perimeter-defense games. IEEE Rob. Autom. Lett. 5, 2738–2745. doi:10.1109/lra.2020.2972818

CrossRef Full Text | Google Scholar

Shishika, D., Maity, D., and Dorothy, M. (2021). “Partial information target defense game,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 30 May–05 June, 2021 (IEEE), 8111–8117.

CrossRef Full Text | Google Scholar

Smith, S. L., Bopardikar, S. D., and Bullo, F. (2009). “A dynamic boundary guarding problem with translating targets,” in Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, Shanghai, China, 15–18 December, 2009 (IEEE), 8543–8548.

CrossRef Full Text | Google Scholar

Tanimoto, S. L., Itai, A., and Rodeh, M. (1978). Some matching problems for bipartite graphs. J. ACM (JACM) 25, 517–525. doi:10.1145/322092.322093

CrossRef Full Text | Google Scholar

Velhal, S., Sundaram, S., and Sundararajan, N. (2022). A decentralized multirobot spatiotemporal multitask assignment approach for perimeter defense. IEEE Trans. Robotics 38, 3085–3096. doi:10.1109/tro.2022.3158198

CrossRef Full Text | Google Scholar

Von Moll, A., Garcia, E., Casbeer, D., Suresh, M., and Swar, S. C. (2020). Multiple-pursuer, single-evader border defense differential game. J. Aerosp. Inf. Syst. 17, 407–416. doi:10.2514/1.i010740

CrossRef Full Text | Google Scholar

Von Moll, A., Shishika, D., Fuchs, Z., and Dorothy, M. (2021). “The turret-runner-penetrator differential game,” in 2021 American Control Conference (ACC), New Orleans, LA, 25–28 May, 2021 (IEEE), 3202–3209.

CrossRef Full Text | Google Scholar

Von Moll, A., Pachter, M., Shishika, D., and Fuchs, Z. (2022a). Circular target defense differential Games$^*$. IEEE Trans. Automatic Control, 1–14. doi:10.1109/tac.2022.3203357

CrossRef Full Text | Google Scholar

Von Moll, A., Shishika, D., Fuchs, Z., and Dorothy, M. (2022b). Turret-runner-penetrator differential game with role selection. IEEE Trans. Aerosp. Electron. Syst. 58, 5687–5702. doi:10.1109/taes.2022.3176599

CrossRef Full Text | Google Scholar

Wang, J., Jin, X., and Tang, Y. (2022). Optimal strategy analysis for adversarial differential games. Electron. Res. Archive 30, 3692–3710. doi:10.3934/era.2022189

CrossRef Full Text | Google Scholar

Yan, R., Shi, Z., and Zhong, Y. (2017). “Escape-avoid games with multiple defenders along a fixed circular orbit,” in 2017 13th IEEE International Conference on Control & Automation (ICCA), Ohrid, Macedonia, 03–06 July, 2017 (IEEE), 958–963.

CrossRef Full Text | Google Scholar

Yan, R., Shi, Z., and Zhong, Y. (2019a). “Construction of the barrier for reach-avoid differential games in three-dimensional space with four equal-speed players,” in 2019 IEEE 58th Conference on Decision and Control (CDC), Nice, France, 11–13 December, 2019 (IEEE), 4067–4072.

CrossRef Full Text | Google Scholar

Yan, R., Shi, Z., and Zhong, Y. (2019b). Reach-avoid games with two defenders and one attacker: An analytical approach. IEEE Trans. Cybern. 49, 1035–1046. doi:10.1109/tcyb.2018.2794769

PubMed Abstract | CrossRef Full Text | Google Scholar

Yan, R., Shi, Z., and Zhong, Y. (2020). Task assignment for multiplayer reach–avoid games in convex domains via analytical barriers. IEEE Trans. Robotics 36, 107–124. doi:10.1109/tro.2019.2935345

CrossRef Full Text | Google Scholar

Yan, R., Shi, Z., and Zhong, Y. (2021a). Cooperative strategies for two-evader-one-pursuer reach-avoid differential games. Int. J. Syst. Sci. 52, 1–19. doi:10.1080/00207721.2021.1872116

CrossRef Full Text | Google Scholar

Yan, R., Shi, Z., and Zhong, Y. (2021b). Optimal strategies for the lifeline differential game with limited lifetime. Int. J. Control 94, 2238–2251. doi:10.1080/00207179.2019.1698770

CrossRef Full Text | Google Scholar

Yan, R., Duan, X., Shi, Z., Zhong, Y., and Bullo, F. (2022). Matching-based capture strategies for 3D heterogeneous multiplayer reach-avoid differential games. Automatica 140, 110207. doi:10.1016/j.automatica.2022.110207

CrossRef Full Text | Google Scholar

Zhou, Z., Takei, R., Huang, H., and Tomlin, C. J. (2012). “A general, open-loop formulation for reach-avoid games,” in 2012 IEEE 51st IEEE conference on decision and control (CDC), Maui, HI, 10–13 December, 2012 (IEEE), 6501–6506.

CrossRef Full Text | Google Scholar