Reinforcement learning for path planning of free-floating space robotic manipulator with collision avoidance and observation noise

Abstract

This study introduces a novel approach for the path planning of a 6-degree-of-freedom free-floating space robotic manipulator, focusing on collision and obstacle avoidance through reinforcement learning. It addresses the challenges of dynamic coupling between the spacecraft and the robotic manipulator, which significantly affects control and precision in the space environment. An innovative reward function is introduced in the reinforcement learning framework to ensure accurate alignment of the manipulator’s end effector with its target, despite disturbances from the spacecraft and the need for obstacle and collision avoidance. A key feature of this study is the use of quaternions for orientation representation to avoid the singularities associated with conventional Euler angles and enhance the training process’ efficiency. Furthermore, the reward function incorporates joint velocity constraints to refine the path planning for the manipulator joints, enabling efficient obstacle and collision avoidance. Another key feature of this study is the inclusion of observation noise in the training process to enhance the robustness of the agent. Results demonstrate that the proposed reward function enables effective exploration of the action space, leading to high precision in achieving the desired objectives. The study provides a solid theoretical foundation for the application of reinforcement learning in complex free-floating space robotic operations and offers insights for future space missions.

1 Introduction

Space exploration represents a sign of human curiosity that drives scientific and technological advancements at the frontier of our knowledge. Space technologies have materialized our wildest dreams, from the exploration of space to the potential colonization of different planets. Among these technologies, robotic manipulators designed for space operations have emerged as critical tools for various missions, including on-orbit servicing, space debris removal, and in-space assembly of large structures due to their high technological readiness level (Papadopoulos et al., 2021). These manipulators perform tasks that would be too dangerous, costly, or time-consuming for human astronauts to perform (Flores-Abad et al., 2014).

Research in space robotic manipulator divides into two distinct categories: free-flying and free-floating (Dubowsky and Papadopoulos, 1993). Free-flying manipulators are characterized by the active control of position and orientation (attitude) of the base spacecraft through the use of thrusters, while the manipulator executes its designated tasks (Seddaoui et al., 2021). This approach, which is analogue to the operation of a fixed-base manipulator, has been extensively studied with focus on path planning and control (Pfeiffer, 1968; Wang et al., 2021). However, the reliance on the thrusters for pose (position and attitude) control incurs significant fuel consumption, potentially reducing the operational lifespan of the system in orbit.

Conversely, free-floating manipulator systems operate with a base spacecraft whose pose is not actively controlled. Instead, the pose of the spacecraft is allowed to naturally react to the manipulator’s joint movements due to the conservation of momentum (Ratajczak and Tchon, 2021; Tsiotras et al., 2023). Intuitively, this approach offers significant advantages, such as the saving of spacecraft fuel and a reduced risk of collision with targets, which might otherwise result from the active attitude control of the base spacecraft. Despite these advantages, controlling free-floating space robots represents a significant challenge. The dynamic interaction between the manipulator and its base spacecraft, due to the conservation of momentum, introduces a complex coupling effect. This effect significantly complicates path planning and motion control, as actions performed by the manipulator can inadvertently after the pose of the base spacecraft, and vice versa. The determination of the pose of the end-effector of the manipulator depends not only on the current joint angles on the manipulator but also on the previous velocities of both the joints and the base spacecraft (Nanos and Papadopoulos, 2017). Additionally, the nonholonomic nature of angular momentum further complicates control strategies, making traditional path-planning solutions, typically used for fixed-base manipulators, ineffective for free-floating space manipulators (Rybus et al., 2017; Dai et al., 2022), highlighting the need for innovative solutions in this area.

2 Related works

To bridge the research gap in the control and path planning of free-floating space manipulator, a challenge not encountered with fixed-base manipulators, various techniques have been proposed (Agrawal et al., 1996; Luo et al., 2018). Early efforts, as outlined in (Agrawal et al., 1996) applied inverse kinematics to path planning while respecting the nonholonomic constraints of angular momentum. However, this approach has limitations in guaranteeing path optimality, as dynamic singularities, which are often unpredictable uniquely from the manipulator’s kinematics, can adversely affect the manipulator’s performance (Xu et al., 2011). In response to these limitations, more recent studies have turned to optimization theory to enhance path-planning solutions, such as particle swarm optimization (Wang et al., 2015) and non-singular terminal sliding mode control (Shao et al., 2021). Despite their promise, these optimization-based solutions often demand substantial computational resources, complicating their application in real-time implementation, especially when facing uncertain disturbances. Additionally, any changes in the initial conditions of the manipulator or its target pose necessitate a comprehensive re-optimization process.

To address this shortfall, researchers have explored the potential of using machine learning (Ye et al., 2019), and more specifically, reinforcement learning (RL), to overcome these limitations (Xie et al., 2020). In essence, RL employs an agent that learns a decision-making policy dynamically through trial-and-error interactions with its environment (in this case, the space manipulator) to maximize a predefined reward function (Sutton and Barto, 2018; Nguyen and Hung, 2019). This learning process, which is within a Markov Decision Process framework, allows the agent to issue commands to alter the environment state and receive feedback in the form of rewards based on the efficacy of its actions. Remarkably, this process does not rely on prior data samples or predefined rules. Hence, the motivation behind exploring the application of RL in tasks like capturing a target by a free-floating space manipulator system lies in its ability to navigate the complexities of the task easily while operating within the constraints of space-based environments. After the training phase concludes, the trained agent is encapsulated solely by a set of weights and biases, represented within neural networks. These parameters are lightweight and can be readily deployed on various hardware platforms without necessitating extensive computational resources. This characteristic is particularly advantageous in space missions where acquiring powerful computational resources is inherently challenging.

Several RL techniques, such as the Deep Deterministic Policy Gradient (DDPG), Actor-Critic algorithm, and Q-learning, have proven effective in path planning for manipulator operating in continuous action spaces. For instance, a study implemented soft Q-learning for path planning of a 3-DOF (Degrees of Freedom) space robotic manipulator (Yan et al., 2018). The reward function in this study was designed based on the distance between the end-effector of the manipulator and the target. It returns positive rewards for minimizing the distance and imposing penalties (negative rewards) for cumulative joint torques.

While the value-based Q-learning performs well when agents select actions from a finite set, policy-gradient based methods like DDPG excel for continuous state and action spaces (Liu and Huang, 2021). In another instance, the DDPG algorithm was applied to path planning for a simple 2-DOF space manipulator (Hu et al., 2018). The reward function in the study included terms for obstacle avoidance, penalties for disturbances from the base spacecraft, and a small penalty for the path length. Additionally, DDPG was used to enhancing the training efficiency of a 3-DOF space robotic manipulator through pre-training (Du et al., 2019). Similarly, a dual-arm space robotic manipulator, each arm with 4-DOF, was trained using DDPG, with the reward function incorporating velocity constraints and penalties to prevent collisions between the arms (Li Haoxuan et al., 2021). Furthermore, the Actor-Critic algorithm was utilized for path planning for a 6-DOF space robotic manipulator, aiming for position and orientation alignment (Liang et al., 2021). The reward function in this case was based not only on the distance difference but also a term for the Euler angle difference between the end effector and the target. In a separate context, DDPG was applied for collision-free path planning for a fixed-base manipulator with 7-DOF, using artificial potential fields within the reward functions to represent obstacles (Li Yinkang et al., 2021). The same algorithm was later applied to a redundant 7-DOF manipulator in a free-floating condition, aiming for target capture with a reward function also based on artificial potential fields (Li et al., 2022).

Table 1 summarizes the current state of RL applications in path planning for space robotic manipulators. This emerging field has seen limited publications to date. The majority of research has focused on training 3-DOF space manipulators, with fewer studies exploring 6-DOF or higher. A significant oversight in many studies in the neglect of joint velocity limitations. Movover, there is a lack of incorporation of quaternion-based expressions for attitude control and consideration of obstacle avoidance in path planning, both of which are crucial for practical applications. Additionally, the use of scalar reward functions in the Markov Decision Process can potentially restrict the search space or result in local optima if not designed with caution. The challenge of balancing exploration-exploitation in RL continues to be a prominent area of research (Jepma and Sander, 2011).

TABLE 1

Ref		Hu et al. (2019)	Yan et al. (2018)	Du et al. (2019)	Li et al. (2021b)	Liang et al. (2021)	Li et al. (2022)
Algorithm		MRDDPG	Soft Q-Learning	DDPG + pre-training	DDPG	Deep RL	DDPG
DOF		2	3	3	4	6	7
Target Capture Constraints	EE* Position
	EE* Attitude	-	-	-	-	(Euler angles)	(Euler angles)
	Joint Velocity	-	-	-		-	-
Obstacle Avoidance		-	-	-	-	-	-
Observation Noise		-	-	-	-	-	-

Summary of current research on RL-based path planning for space manipulators.

* EE, End Effector.

The development of RL path planning in free-floating space manipulators remains scarce. Research suggests that constructing an effective reward function capable of optimizing the control strategy of an agent in a perturbed environment is still a significant challenge.

This paper proposes a deep RL algorithm that integrates DDPG and Actor-Critic neural networks for efficient path planning for a 6-DOF free-floating space manipulator. Our methodology aims to expedite training convergence in dynamic settings by designing reward functions that consider several critical factors. These include aligning the manipulator’s end effector (EE) with the target using a novel quaternion-based approach for orientation comparison and incorporating joint velocity constraints to achieve smoother EE trajectories. Additionally, our algorithm introduces measures to prevent self-collision of the manipulator’s links and, importantly, incorporates external obstacle avoidance capabilities during target approach - a novel feature not previously implemented in the literature. Recognizing the imperfection of real-world sensors, we further enhance our model’s practical applicability by including white noise in the observations during training. This ensures that the trained agent is resilient to signal noise and capable of achieving successful target capture. The effectiveness and advantages of our proposed algorithm are validated through numerical simulations.

3 Mathematic model of a free-floating space manipulator

A general free-floating space robotic manipulator is composed of an n-DOF manipulator mounted on a floating base spacecraft, as shown in Figure 1. An inertial reference frame, designated as OXYZ, is defined in space. Within this frame, the base spacecraft (B₀) is treated as link 0 with 6-DOF and the first link of the manipulator is affixed to the base spacecraft at joint J₁. The manipulator itself is constructed from n rigid links connected in series by n independently actuated revolute joints. The position of the EE is denoted by a vector .

FIGURE 1

The position () and orientation () of the EE can be accurately determined by propagating the relative position and orientation of each joint from the base spacecraft to the end-effector, such thatwhere is the position vector of the ith joint in the inertial frame and is the rotation matrix of the ith link. Differentiating Eqs 1, 2 with respect to time results in the linear () and angular () velocities of the EE, such thatwhere and are the ith joint’s angular velocity and direction in the inertial frame, respectively. Rewriting the velocity equations into a compact matrix form with the base state and EE state yieldswhere is the joint angular velocity vector, and (, ) are the Jacobian matrices dependent on the base spacecraft and manipulator motions, respectively.

In a free-floating space manipulator system, the total momentum is conserved. Assuming that the initial momenta of the system is zero, we can express the conservation of momentum conservation asor in the compact matric form aswhere is the mass of the ith link, is the inertial moment of the ith link at its center of mass (CM), are fractional coefficients measuring the distances from the CM of the ith link to its two ends and satisfy the condition of , is the inertia matrix of the base spacecraft, and is the coupling inertia matrix of the manipulator. The detailed expression of these inertia matrices can be found in Wilde et al. (2018).

Combining Eqs. 5, 8 yieldswhere is the generalized Jacobian matrix dependent on the base spacecraft attitude, joint angles, and mass properties of the base and links.

Based on the above kinematics, the dynamics of the free-floating n-link robotic manipulator system, as shown in Figure 1, can be derived by Lagrange equations (Wilde et al., 2018).whereand is the generalized displacement vector of the free-floating space robotic manipulator, is the inertia matrix reflecting the dynamic coupling between the base spacecraft and the manipulator, and represents the Coriolis forces acting on the base spacecraft and the manipulator, and is the control torque at the joints. This equation serves as the mathematical base in our dynamic simulation environment of a free-floating manipulator, which will be used for reinforcement learning training.

To ascertain the algorithm’s performance and effectiveness under real-world conditions, the parameters used in the simulations are based on actual space robotic manipulator missions (Li et al., 2019). Table 2 shows a brief summary of past space robotic manipulator missions. Notably the mass ratios in most cases are small except for the Orbital Express mission, with a 7.4% mass ratio being the highest. Accordingly, this mass ratio was selected in study to show the need to consider the free-floating base condition.

4 Reinforcement learning for path planning

4.1 DDPG reinforcement learning algorithm

The control objective of the free-floating manipulator is to move the EE smoothly towards a specified target with precise position and orientation without any collisions between the manipulator’s links, the target, and any obstacles along the path. With this in mind, the observations that are critical to the learning process are defined aswhere subscripts “T”, “e”, and “Ob” refer to the target, the end-effector, and Obstacle, respectively, and are the distance and orientation difference between the EE and the target, respectively, to be described in details in Section 4.

The actions that control the manipulator are the torque input at each joint:

Assuming that the continuous system dynamics between states and the actions are discretized in the time domain at time step t_i, we obtain:where is the control input applied to the system and is the noise representing the unmodeled system dynamics in the environment.

Applying RL algorithms, such as Q-learning, to a continuous system like robotic path planning is challenging. This challenge arises from the necessity of optimizing actions at each timestep i through a greedy policy. An alternative, the Deep Q-Network (DQN) algorithm (Silver et al., 2014) offers a solution by estimating the Q function with deep neural networks (DNNs) that take states and actions as inputs and return the expected cumulative reward as output. The DQN algorithm uses two distinct DNNs, the main network and target, which operate concurrently. DQN is able to learn value functions using in a stable and robust way due to two innovations (Lillicrap et al., 2015). The network is trained off-policy with samples from a replay buffer to minimize correlations between samples, and the network is trained with a target Q network to give consistent targets during temporal difference backups. This is also used in the DDPG algorithm. Moreover, the response to the input actions by the robotic manipulator is deterministic. Thus, instead of learning a set of probability distributions, a deterministic action can be directly learned from given states for the robotic manipulator system (Silver et al., 2014). Accordingly, the DDPG algorithm is used here to compute the policy gradient .

The stability of policy gradient methods has been well-studied under the Lyapunov framework. For instance, Hejase and Ozguner, (2023) proved that the policy gradient is negative statistically, which is a crucial factor in ensuring the stability of DDPG methods from a statistical standpoint.

In the DDPG, the actor network deterministically returns in a continuous action space based on its interaction with the environment by the parameterized policy and is updated by the application of gradient ascent on the policy gradient,where N is the time steps in each episode, is a deterministic policy, and is a behavior policy with different trajectories from the policy trajectories . The deterministic policy gradient can be calculated by trajectories that are sampled from a distinct behavior policy based on an Actor-Critic approach (Degris et al., 2012),

Here, is the Q-function that represents the expected discounted return calculated by starting from the ith time step, taking the state-action pair and generating rewards based on the policy until the end of the episode. Notably, the Q-function depends exclusively on environment, meaning that it can be learned off policy by the Q-learning algorithm (Watkins and Peter, 1992). The parameter is a discount factor. If , the agent treats all future rewards equally, which is suitable for long-term planning. On the opposite, the agent is for short-term planning if .

The critic network evaluates the agent’s performance by estimating the Q-value of state-action pairs . The Q-function is approximated by the Bellman equation (Bellman, 1996).and using Temporal difference, the Q-function can be calculated recursively over a parameter , to guide the next-generation of actions.

The critic parameters are updated by reducing the loss function, such that,

To ensure the samples are independently distributed, a replay buffer is created to store the tuple that are sampled from the environment according to the exploration policy (Lillicrap et al., 2015). At each time step, the actor and critic are updated by sampling a minibatch uniformly from the buffer. When the replay buffer is full the oldest samples are discarded. Because DDPG is an off-policy algorithm, the replay buffer can be large, allowing the algorithm to benefit from learning across a set of uncorrelated transitions.

Directly implementing Q learning with neural networks proved to be unstable in many environments. However, for DDPG the target actor and critic parameters are soft updated, rather than directly copying the weights. The weights of these target networks are updated by having them slowly track the learned networks such that and to improve the Q-learning stability, where is the smoothing factor (Lillicrap et al., 2015). This means that the target values are constrained to change slowly, greatly improving the stability of learning. This simple change moves the relatively unstable problem of learning the action-value function closer to the case of supervised learning. The pseudo-code that describes the detailed training process for the DDPG algorithm is shown in Table 3.

4.2 Actor-Critic networks

The actor and critic DNNs are created using standard feed-forward neural networks, each one with two hidden layers. The first and second hidden layers have 400 and 300 neurons, respectively. The actor network inputs are the states of the system, and its outputs are the actions. However, the critic network inputs are the state-action pairs, and its outputs are the action values. In this study, the robotic manipulator includes 6 links, i.e., n = 6. Thus, the dimensions of the state and actions are , and , respectively.

4.3 Reward function

As the most critical part of reinforcement learning, the reward function is directly responsible for how the algorithm trains and what objectives are achieved. With that in mind, an innovative reward function is developed to minimize the pose difference between the EE and the target as well as the sudden changes in joint velocities. The reward function () is evaluated at each time step, and the sum of these evaluations across all time steps constitutes the episode reward.

The pose difference is represented by the position error d and the orientation error . The primary part of the reward function must include:where and are positive reward coefficients. As the pose difference is minimized, the reward is maximized. Notably this guidance term will ensure the agent is learning to align the EE and the target.

The position error is determined straightforward by:

The orientation error of the EE can be defined either by a unit quaternion () or a set of Euler angles . In the previous work as shown in Table 1, the expression of the orientation error is based on a set of Euler angles, such that,where and are the Euler angles of the EE and the target, respectively. It is worth pointing out that the orientation difference in such a form loses the information of rotation sequence of Euler angles. This increases the search space in the learning process and may even lead to divergence. Thus, the Euler angle representation is not used in the current work. Instead, an innovative reward function is built using the quaternion representation of orientations.

Let the orientations of the EE and the target be represented by two unit quaternions and , respectively, as shown in Figure 2.

FIGURE 2

The orientation error of the EE with respect to the target can be expressed as (Campa and Camarillo, 2008):where is the conjugate of the quaternion . Accordingly, an orientation difference between the EE and the target is determined by:

As the EE approaches the desired pose, the pose error (, ) diminishes as well as the reward function in Eq. 20. To ensure the EE stays within the desired pose for capture, large rewards (, ) are added to the reward function in Eq. 20 if and are within predefined ranges, such that:where and are pre-defined indicators for close proximity operation region for position and orientation, respectively. Once the end-effector enters into this region, large positive rewards are assigned to the reward function to encourage and accelerate the agent converging towards the desired pose.

Furthermore, a large negative penalty () is added to the reward function in Eq. 25 to avoid self-collision between the robotic links and the target. If the distance between any two is less than a predefined safe distance, the penalty is added to the reward function, such that:where is a penalty for two links being too close to each other.

To facilitate obstacle avoidance, a substantial negative penalty () is incorporated into the reward function in Eq. 26. This penalty () is activated whenever the distance between the obstacle’s CM and any link’s CM falls below a predefined safety range:

Figure 3 shows a simplified illustration of how the safety distance defined between any link and the obstacle is defined.

FIGURE 3

Finally, in order to ensure a smooth path and avoid sudden changes in motion, a velocity penalty is incorporated as a negative reward () into the reward function described in Eq. 27. This leads to the final reward function R_t,where is a velocity penalty for smooth joint velocities by multiplying the squared sum of joint angular velocities.

In summary, the core aspect of RL lies in crafting a robust reward function. The adaptability of our approach shines as we can tailor the reward terms in the reward function to suit varied missions and tasks. This adaptability is evidently demonstrated by the above process to augment the reward function with multiple terms—comprising rewards or penalties—that correspond to the specific objectives at hand. This iterative process ensures our algorithm remains versatile and effective across diverse scenarios.

5 Simulation outcomes and analysis

5.1 Simulation environment

In this study, the proposed RL approach for motion planning of a free-floating space robotic manipulator is verified by simulation using MATLAB^® and Simulink^®. The physical parameters of the manipulator are given in Table 4. Table 5 provides the RL training hyperparameters and the initial conditions for all simulation cases. These simulations were performed on a PC equipped with an Intel^® Core™ i7-10600K Processor and 32 GB of RAM.

5.2 Simulation results

Three different cases of path planning were conducted to demonstrate the capabilities of the proposed RL approach, each employing different reward functions. The first case used the reward function in Eq. 27 without velocity constraints. The second case used the reward function in Eq. 28 with the velocity constraints. The third case used the same reward function as the second case. However, it included noisy observations by adding white noise, thereby closely reflecting realistic sensor observations.

5.2.1 Case 1–approach target without velocity constraints

In this case, the free-floating space robotic manipulator has been trained by the reward function in Eq. 27 toward the desired pose . The parameters in this reward function are defined as , , , , , , , . These parameters are used in all simulation cases described below.

The training progress is shown in Figure 4, where the blue line is the cumulative rewards from all timesteps in each episode, and the red line is the moving average of the cumulative rewards of all episodes. These curves show that the agent learns to reach the desired pose by searching for the highest reward. The term () in the reward function efficiently guides the EE towards the capture range within the first 5,000 episodes, while the term () in the reward function effectively prevents collisions of the manipulator with the obstacle, the target and itself.

FIGURE 4

Upon entering the capture range, the large rewards () in the reward function are activated to keep the EE within the capture range while simultaneously refining the EE’s pose throughout the remaining timesteps to prepare the gripper for capture. The large variation of the episode reward reflects the activation and deactivation of the large rewards () in this process. Figure 5 shows several snapshots of the output path for the space robotic manipulator towards the target.

FIGURE 5

Figure 6A shows the pose error between the EE and the target over time with the left subfigure representing the distance error () and the right subfigure depicting the orientation difference (). The trained agent guides the EE towards the desired target pose range (, ) within approximately 6 s. Subsequently, the EE stays within that region to achieve capture for the remaining duration of the simulation. It is worth mentioning that the threshold () does not imply that the minimum distance error the trained agent can reach; instead, it signifies that crossing this threshold secures a large positive reward (). With this large positive reward, the agent achieves a minimum distance error of Δd = 0.058 m and orientation error of Δθ = 4.85°.

FIGURE 6

Figure 6B shows the time histories of actions (control torques) at all 6 joints as given by the agent. The agent effectively provides the necessary torques to achieve a valid path for the manipulator to move the EE towards the target, ensuring no self-collision occurs between manipulator links, and most importantly, avoiding the obstacle placed between the EE and the target.

It is worth pointing out that the RL hyperparameters play a pivotal role in governing various aspects of the learning process, including learning dynamics, exploration versus exploitation trade-offs, convergence, adaptation to changing environments, and computational resource utilization. The training is greatly influenced by the learning rate, replay buffer and discount factor. The learning rate dictates the pace at which the RL agent updates its estimates based on new information, influencing both the speed and stability of learning. Similarly, a larger replay buffer enhances sample efficiency and stability by enabling the agent to learn from a diverse array of past experiences while mitigating overfitting risks. Moreover, the discount factor is essential for the agent’s long-term decision-making, allowing it to balance immediate rewards with future gains. This factor significantly influences the convergence and stability of RL algorithms and aids in handling uncertainty and delayed rewards, thus facilitating effective exploration and exploitation strategies.

5.2.2 Case 2–approach target with velocity constraints

To achieve smoother motion and eliminate undesired sudden changes in the path, as shown in Figure 6A at the beginning of the path, often associated with high joint angular velocities, a velocity penalty term for all six joint velocities () has been added to the reward function, as shown in Eq. 27. The parameters for this simulation case are the same as those in case 1, with the inclusion of an additional parameter of .

The training results are shown in Figures 7, 8. Figure 7 shows the convergence of the reward function. Figure 8A shows the pose error between the EE and the target over time, and Figure 8B shows the time histories of actions (control torques) for all 6 joints given by the agent.

FIGURE 7

FIGURE 8

In comparison to Figures 4, 7 shows that the training process is similar to Case 1. The training converges to the capture range in approximately 5,000 episodes. Upon entering the capture range, the large rewards () in the reward function are activated to keep the EE within the capture range while simultaneously refining the EE’s pose throughout the remaining timesteps to prepare the gripper for capture. However, it is noted that in Case 2, the introduction of a velocity penalty term () to the reward function results in a smoother path. Comparing Figure 8A with Figure 6A reveals an improvement in the convergence of the path in the first 3 s. Furthermore, the end-effector successfully converges towards the capture range. The high rewards () helps the agent staying in this region for future capture and servicing with a minimum distance error of Δd = 0.088 m and orientation difference of Δθ = 6.61°.

To further examine the effect of the velocity constraint term on agent training, the square sum of all six joint velocities () along the path is shown in Figure 9. It is observed that the square sum of velocities obtained from the path in Case 2 (right plot) barely surpasses 80 in comparison to the 160 reached in Case 1 (left plot). These results clearly demonstrate the efficacy of modifying the reward function on the solution. Such findings open the door for future research to enhance path optimization by designing better reward functions, potentially incorporating torque penalties, energy consumption penalties, and the integration of optimal path constraints commonly used in optimization to RL.

FIGURE 9

5.2.3 Case 3–approach target with noisy observations

In real missions, assuming perfect observations of the state is unrealistic, particularly in estimating the target’s pose using vision sensors like cameras. To evaluate the robustness of the proposed reward function in handling noisy observations, training of the agent was proceeded with the same reward function and parameters in Case 2, except introducing white noise to the observations that would be obtained from cameras in real mission. The noise is added to the target pose and obstacle position: with a signal-to-noise ratio (SNR) of 10/1 to simulate realistic conditions:where is a random number with Mean = 0 and Variance = 1.

The training outcome is shown in Figure 10. It is noted that the agent is able to obtain high rewards despite the presence of observational noise. Figure 11A shows the pose error between the EE and the target over time. Despite the presence of noise in the target position and orientation observations, the agent effectively manages to align the EE and the target for capture. The agent achieves a minimum distance error of Δd = 0.048 m and orientation difference of Δθ = 7.79°. Notably, as the EE approaches the target, the absolute value of the noise decreases, imitating the characteristic behavior observed in real camera observations. The agent successfully overcomes the challenge of noisy inputs, which proves the effectiveness and robustness of the proposed solution.

FIGURE 10

FIGURE 11

Figure 11B shows the time histories of actions (control torques) for all six joints given by the agent. Observations reveal that the torques required for capture exhibit more oscillations compared to the previous solution in Figure 8. This increase in oscillation results from the noisy observations that add uncertainties during training. It is worth mentioning that the agent trained in Case 3 (noise on observations) still manages to achieve the desired task when implemented in a noise-free environment. Conversely, the agent trained in Case 2 (no noise on observations) fails its task in a noisy environment. This highlights the importance of RL training under realistic and challenging conditions. Exposing the agent to noise during training equips it with the ability to effectively achieve its task in real-world applications.

6 Conclusion

This study investigated the obstacle avoidance path planning problem for a free-floating space robotic manipulator using RL. The DDPG algorithm was used to train an agent to control a 6-DOF space robotic manipulator to achieve the desired goals. Specifically, it aims to find a feasible path towards position and orientation alignment between the EE and the target, without any self-collation between manipulator links, while avoiding external obstacles along the path. The proposed method was verified using simulation with three different simulation cases–without joint velocity constraints, with joint velocity constraints, and with noisy observations. The results show the proper construction of a reward function is critical for training the agent using RL and affects the quality of the path. The inclusion of observation noise in the training will significantly enhance the robustness of the agent. The offline nature of the training in the RL approaches diminishes the need for immediate computational efficiency considerations. After the training phase concludes, the trained agent is encapsulated solely by a set of weights and biases, represented within neural networks. These parameters are lightweight and can be readily deployed on various hardware platforms without necessitating extensive computational resources. This characteristic is particularly advantageous in space missions where acquiring powerful computational resources is inherently challenging.

References

• 1

  AgrawalS. K.GarimellaR.DesmierG. (1996). Free-floating closed-chain planar robots: kinematics and path planning. Nonlinear Dyn.9, 1–19. 10.1007/BF01833290

  • CrossRef
  • Google Scholar

• 2

  ASC-CSA (2024a). Canadarm, Canadarm2, and Canadarm3 – a comparative table. Available at: https://www.asc-csa.gc.ca/eng/iss/canadarm2/canadarm-canadarm2-canadarm3-comparative-table.asp April 5, 2024).

  • Google Scholar
  • View reference in article

• 3

  ASC-CSA (2024b). Dextre. Available at: https://www.asc-csa.gc.ca/eng/iss/dextre/data-sheet.asp April 5, 2024).

  • Google Scholar
  • View reference in article

• 4

  BellmanR. (1966). Dynamic programming. Science153, 34–37. 10.1126/science.153.3731.34

  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  CampaR.CamarilloK. (2008). Unit quaternions: a mathematical tool for modeling, path planning and control of robot manipulators. Robot. Manip. InTech. 10.5772/6197

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  DaiYeXiangC.ZhangY.JiangY.QuW.ZhangQ. (2022). A Review of spatial robotic arm trajectory planning. Aerospace9 (7), 361. 10.3390/aerospace9070361

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  DegrisT.WhiteM.SuttonR. S. (2012). Off-policy actor-critic. Available at: https://arxiv.org/abs/1205.4839.

  • Google Scholar
  • View reference in article

• 8

  DuD.ZhouQ.QiN.WangXuLiuY. (2019). “Learning to control a free-floating space robot using deep reinforcement learning,” in 2019 IEEE International Conference on Unmanned Systems (ICUS), Beijing, China, October, 2019, 519–523.

  • Google Scholar
  • View reference in article

• 9

  DubowskyS.PapadopoulosE. (1993). The kinematics, dynamics, and control of free-flying and free-floating space robotic systems. IEEE Trans. robotics automation9 (5), 531–543. 10.1109/70.258046

  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  ETS-VII (2024). ETS-VII. Available at: https://www.eoportal.org/satellite-missions/ets-vii#background-on-ets-missions April 5, 2024).

  • Google Scholar
  • View reference in article

• 11

  European Robotic Arm (2024). European robotic arm. Available at: https://www.esa.int/Science_Exploration/Human_and_Robotic_Exploration/International_Space_Station/European_Robotic_Arm April 5, 2024).

  • Google Scholar
  • View reference in article

• 12

  Flores-AbadA.MaOuPhamK.UlrichS. (2014). A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci.68, 1–26. 10.1016/j.paerosci.2014.03.002

  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  HejaseB.OzgunerU. (2023). “Lyapunov stability regulation of deep reinforcement learning control with application to automated driving,” in 2023 American Control Conference (ACC), San Diego, CA, USA, May, 2023, 4437–4442.

  • Google Scholar
  • View reference in article

• 14

  HuX.HuangX.HuT.ZhongS.HuiJ. (2018). “Mrddpg algorithms for path planning of free-floating space robot,” in 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, November, 2018, 1079–1082.

  • Google Scholar
  • View reference in article

• 15

  JEMRMS (2024). JEMRMS. Available at: https://iss.jaxa.jp/en/kibo/about/kibo/rms/April 5, 2024).

  • Google Scholar
  • View reference in article

• 16

  JepmaM.SanderN. (2011). Pupil diameter predicts changes in the exploration–exploitation trade-off: evidence for the adaptive gain theory. J. cognitive Neurosci.23 (7), 1587–1596. 10.1162/jocn.2010.21548

  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  LiH.GongD.YuJ. (2021a). An obstacles avoidance method for serial manipulator based on reinforcement learning and Artificial Potential Field. Int. J. Intelligent Robotics Appl.5, 186–202. 10.1007/s41315-021-00172-5

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  LiW.-J.ChengD.-YiLiuX.-G.WangY.-B.ShiW.-H.TangZ.-X.et al (2019). On-orbit service (OOS) of spacecraft: a review of engineering developments. Prog. Aerosp. Sci.108, 32–120. 10.1016/j.paerosci.2019.01.004

  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  LiY.LiD.ZhuW.SunJ.ZhangX.LiS. (2022). Constrained motion planning of 7-DOF space manipulator via deep reinforcement learning combined with artificial potential field. Aerospace9 (3), 163. 10.3390/aerospace9030163

  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  LiY.XiaolongH.SheY.LiS.YuM. (2021b). Constrained motion planning of free-float dual-arm space manipulator via deep reinforcement learning. Aerosp. Sci. Technol.109, 106446. 10.1016/j.ast.2020.106446

  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  LiangB.ChenZ.GuoM.WangY.WangY. (2021). Space robot target intelligent capture system based on deep reinforcement learning model. J. Phys. Conf. Ser.1848 (1), 012078. 10.1088/1742-6596/1848/1/012078

  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  LillicrapT. P.HuntJ. J.PritzelA.HeessN.ErezT.TassaY.et al (2015). Continuous control with deep reinforcement learning. Available at: https://arxiv.org/abs/1509.02971.

  • Google Scholar
  • View reference in article

• 23

  LiuY.-C.HuangC.-Yu (2021). DDPG-based adaptive robust tracking control for aerial manipulators with decoupling approach. IEEE Trans. Cybern.52 (8), 8258–8271. 10.1109/TCYB.2021.3049555

  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  LuoJ.YuM.WangM.YuanJ. (2018). A fast trajectory planning framework with task-priority for space robot. Acta Astronaut.152, 823–835. 10.1016/j.actaastro.2018.09.023

  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  NanosK.PapadopoulosE. G. (2017). On the dynamics and control of free-floating space manipulator systems in the presence of angular momentum. Front. Robotics AI4, 26. 10.3389/frobt.2017.00026

  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  NguyenH.HungLa. (2019). “Review of deep reinforcement learning for robot manipulation,” in 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy, February, 2019, 590–595.

  • Google Scholar
  • View reference in article

• 27

  Orbital Express (2024). Orbital express. Available at: https://mda.space/en/orbital-express/April 5, 2024).

  • Google Scholar

• 28

  PapadopoulosE.AghiliF.MaOuLamparielloR. (2021). Robotic manipulation and capture in space: a survey. Front. Robotics AI8, 686723. 10.3389/frobt.2021.686723

  • CrossRef
  • Google Scholar

• 29

  PfeifferF. (1986). Manipulator trajectory planning and control. IFAC Proc. Vol.19 (14), 325–330. 10.1016/s1474-6670(17)59499-9

  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  RatajczakJ.TchońK. (2021). Coordinate-free jacobian motion planning: a 3-d space robot. IEEE Trans. Syst. Man, Cybern. Syst.52 (8), 5354–5361. 10.1109/tsmc.2021.3125276

  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  RybusT.SewerynK.SasiadekJ. Z. (2017). Control system for free-floating space manipulator based on nonlinear model predictive control (NMPC). J. Intelligent Robotic Syst.85, 491–509. 10.1007/s10846-016-0396-2

  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  SeddaouiA.Mini SaajC.NairM. H. (2021). Modeling a controlled-floating space robot for in-space services: a beginner’s tutorial. Front. Robotics AI8, 725333. 10.3389/frobt.2021.725333

  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  ShaoX.SunG.XueC.LiX. (2021). Nonsingular terminal sliding mode control for free-floating space manipulator with disturbance. Acta Astronaut.181, 396–404. 10.1016/j.actaastro.2021.01.038

  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  SilverD.LeverG.HeessN.DegrisT.WierstraD.RiedmillerM. (2014). “Deterministic policy gradient algorithms,” in International conference on machine learning, Beijing, China, June, 2014, 387–395.

  • Google Scholar
  • View reference in article

• 35

  SuttonR. S.BartoA. G. (2018) Reinforcement learning: an introduction. Cambridge, MA, USA: MIT Press, 2–25.

  • Google Scholar
  • View reference in article

• 36

  TsiotrasP.King-SmithM.TicozziL. (2023). Spacecraft-mounted robotics. Annu. Rev. Control, Robotics, Aut. Syst.6, 335–362. 10.1146/annurev-control-062122-082114

  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  WangL.LaiX.MengQ.WuM. (2021). Effective control method based on trajectory optimization for three-link vertical underactuated manipulators with only one active joint. IEEE Trans. Cybern.53, 3782–3793. 10.1109/tcyb.2021.3125187

  • CrossRef
  • Google Scholar
  • View reference in article

• 38

  WangM.LuoJ.UlrichW. (2015). Trajectory planning of free-floating space robot using Particle Swarm Optimization (PSO). Acta Astronaut.112, 77–88. 10.1016/j.actaastro.2015.03.008

  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  WatkinsC.PeterD. (1992). Q-learning. Mach. Learn.8, 279–292. 10.1023/a:1022676722315

  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  WildeM.Kwok ChoonS.GromponeA.RomanoM. (2018). Equations of motion of free-floating spacecraft-manipulator systems: an engineer's tutorial. Front. Robotics AI5, 41. 10.3389/frobt.2018.00041

  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  XieZ.SunT.KwanT. H.MuZ.WuX. (2020). A new reinforcement learning based adaptive sliding mode control scheme for free-floating space robotic manipulator. IEEE Access8, 127048–127064. 10.1109/ACCESS.2020.3008399

  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  XuW.LiangB.XuY. (2011). Practical approaches to handle the singularities of a wrist-partitioned space manipulator. Acta Astronaut.68 (1-2), 269–300. 10.1016/j.actaastro.2010.07.004

  • CrossRef
  • Google Scholar
  • View reference in article

• 43

  YanC.ZhangQ.LiuZ.WangX.LiangB. (2018). “Control of free-floating space robots to capture targets using soft q-learning,” in 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia, December, 2018, 654–660.

  • Google Scholar
  • View reference in article

• 44

  YeX.DongZ.-H.Jia-CaiH. (2019). Research on adaptive reaction null space planning and control strategy based on VFF–rls and SSADE–ELM algorithm for free-floating space robot. Electronics8 (10), 1111. 10.3390/electronics8101111

  • CrossRef
  • Google Scholar
  • View reference in article