A lower extremity robotic exoskeleton device (Androwis et al., 2017) is currently under development by the authors to assist patients with ADL, such as balance, ambulation and gait rehabilitation. In Figure 1A, the physical prototype of this exoskeleton is shown. The total mass of the exoskeleton is 20.4 k g and the frame of the exoskeleton has been manufactured with Onyx (Markforged’s nylon with chopped fiber) reinforced by continuous carbon fiber between layers, using Markforged’s Mark Two printer (Markforged, INC., MA). The exoskeleton has 14 independent DoFs (including six global DoFs for the pelvis root joint): one DoF for the hip flexion/extension joint, one DoF for the knee flexion/extension joint, and two DoFs for the ankle dorsiflexion/plantarflexion and inversion/eversion joint on each side of the body. The 6-DoF pelvis (root) joint is a free joint and unactuated, and the rest eight DoFs on both sides are actuated. Unlike most commercial rehabilitation exoskeletons that have either passive or fixed ankles, the ankle of our system includes powered 2-DoF to assist with dorsiflexion/plantarflexion and inversion/eversion (Nunez et al., 2017). All joints of the robotic exoskeleton are powered by smart actuators (Dynamixel Pro Motor H54-200-S500-R).

Both hip and knee joints are driven by bevel gears for compact design. The ankle is actuated by two parallel motors that are attached to the posterior side the shank support and operate simultaneously in a closed-loop to flex or abduct the ankle. Figure 1C shows the foot model of our exoskeleton system. At the bottom of each foot plate, four 2000N 3-axis force transducers (OptoForce Kft, Hungary) are installed to measure GRFs. These measured forces can be used to determine the CoP in real-time. The CoP is the point on the ground where the tipping moment acting on the foot equals zero (Sardain and Bessonnet, 2004), with the tipping moment defined as the moment component that is tangential to the foot support surface. Let O i denote the location of the i − t h force sensor and F i be its measured force, n denote the unit normal vector of the foot support surface, and C denote the CoP point where the tipping moment vanishes: [ ∑ i 4 ( O i C × F i ) ] × n = 0 (1) from which C can be computed.

A multibody dynamics model of the exoskeleton, shown in Figure 1A, is created with mass and inertia properties estimated from each part’s 3D geometry and density or measured mass. Simple revolute joints are used for the hips and knees. Each ankle has two independent rotational DoFs but their rotation axes are located at different positions (Figure 1C). These two DoFs are physically driven by the closed-loop of two ankle motors together with linkage of universal joints and screw joints. When extending or flexing the ankle, both motors work in sync to move the two long screws up or down simultaneously. When adducting or abducting the ankle, the motors move the screws in opposite directions. The motors can generate up to 160 N m dorsi/plantar flexion torque (Nunez et al., 2017). In this study, we do not directly control these ankle motors. Instead, we assume the two ankle DoFs can be independently controlled with sufficient torques from these motors.

The controller (or control policy) is learned through a continuous RL process. We design the control policy through a neural network with parameters θ, denoting the weights and bias in the neural network. The control policy can be expressed as π θ ( a | s ) and the agent (neural network model) learns to update the parameters θ to achieve the maximum reward. The learning controller (i.e., control policy network) is implemented as a Multi-Layer Perception (MLP) network that consists of three fully connected layers and ReLU as the activation function, as illustrated in Figure 2, the sizes of three layers are set to 256, 256 and 128, respectively. At every time step t, the agent observes the state s t from the environment, and selects an action a t according to its control policy π ( a t | s t ) . The control policy network π ( a | s ) is in the form of the probability distribution of actions in a given state. This action distribution is modeled as a Gaussian, with a state dependent mean μ ( s ) defined by the network, and a standard deviation (STD) Σ that is learned as the neural network parameters: π ( a | s ) = N ( μ ( s ) , Σ ) (4) The action a t is sampled from this distribution, and then converted to control commands that drive the exoskeleton in the environment, which results a new state s t + 1 and a scalar reward r t immediately. The objective is to learn a control policy that maximizes the discounted sum of reward: π * = arg max π E τ ∼ p ( τ | π ) [ ∑ t = 0 T − 1 γ t r t ] (5) where γ ∈ ( 0,1 ) is the discount factor, τ is the trajectory { ( s 0 , a 0 , r 0 ) , ( s 1 , a 1 , r 1 ) , ⋯ } and p ( τ | π ) denotes the likelihood of a trajectory τ under a given control policy π, and T is the horizon of an episode.

The control policy is task- or motion-specific and starts with motion imitation. Within the learning process (Figure 2), the input of the control policy network is defined by s = { p t − 2 : t , v t − 2 : t , c t − 2 : t , a t − 3 : t − 1 , p ^ t + 1 : t + 6 } , in which p and v are joint positions and velocities of the exoskeleton, and a t − 3 : t − 1 represents the action history of three previous steps. To learn a particular skill, we utilize the corresponding target joint pose from the task motion at six future time-steps p ^ t + 1 : t + 6 as the motion prior for feasible control strategies. Considering the importance of CoP as an indicator of system balance and its ready availability from our exoskeleton design, we incorporate the CoP position history c t − 2 : t into the state as a feedback to the controller. As summarized in Figure 2, the learning controller is given as input the combination of the state history, the action history and the future target motions and outputs the joint target positions as the actions. The use of task motion data alleviates the need to design task-specific reward functions and thereby facilitates a general framework to learn a diverse array of behaviors.

To train the control policy network, we use a model-free RL algorithm known as Proximal Policy Optimization (PPO). An effective solution to many RL problems is the family of policy gradient algorithms, in which the gradient of the expected return with respect to the policy parameters is computed and used to update the parameters θ through gradient ascent. PPO is a model-free policy gradient algorithm that samples data through interaction with the environment and optimizes a “surrogate” objective function (Schulman et al., 2017). It utilizes a trust region constraint to force the control policy update to ensure that the new policy is not too far away from the old policy. The probability ratio r t ( θ ) is defined by: r t ( θ ) = π θ ( a t | s t ) π θ o l d ( a t | s t ) . (6) This probability ratio is a measure of how different the current policy is from the old policy π θ o l d (the policy before the last update). A large value of this ratio means that there is a large change in the updated policy compared to the old one. PPO also introduces a modified objective function that adopts clipped probability ratios which forms a pessimistic estimate of the policy’s performance and avoids a reduction in performance during the training process. The “surrogate” objective function is described by considering the clipped objective: L ( θ ) = E t [ m i n ( r t ( θ ) A ^ t , c l i p ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ] (7) where ϵ is a small positive constant which constrain the probability ratio r t ( θ ) . A ^ t denotes the advantage value at time step t. c l i p ( ⋅ ) is the clipping function. Clipping the probability ratio discourages the policy from changing too much and taking the minimum results in using the lower, pessimistic bound of the unclipped objective. Thus any change in the probability ratio is included when it makes the objective worse, and otherwise is ignored. This can prevent the policy from changing too quickly and leads to more stable learning. The control policy can be updated by maximizing the clipped discounted total reward in Eq. 7 with a gradient ascent.

It has been reported that the performance of RL for continuous control depends on the choice of action types (Peng and van de Panne, 2017). It works better when the control policy outputs PD position targets rather than joint torque directly. In this study, we also let the learning controller output the joint target positions as actions (shown in Figure 2). To obtain smooth motions, actions from the control policy network are first processed by a second low-pass filter before being applied to the robot. Our learning process allows the control policy network and the environment to operate at different frequencies since the environment often requires a small time step for integration. During each time integration step, we apply preprocessed actions that are linear interpolated from two consecutive filtered actions. Then the preprocessed actions are specified as PD targets and the final PD-based torques applied to each joint are calculated as τ = c l i p ( k p ( a t − p t ) − k v p ˙ t , − τ ^ , τ ^ ) (8) where k p and k v are proportional gain and differential gain, respectively. The function c l i p ( ⋅ ) returns the upper bound τ ^ or the lower bound − τ ^ if the torque τ exceeds the limit.

We design the reward function to encourage the control policy to imitate a target joint motion p ^ t of the exoskeleton while maintaining balance with robustness. The reward function consists of pose reward r t p , velocity reward r t v , end-effector reward r t e , root reward r t r o o t , center of mass reward r t c o m , CoP reward r t c o p , and torque reward r t t o r q u e , which is defined by: r t = w p r t p + w v r t v + w e r t e + w r o o t r t r o o t + w c o m r t c o m (9) + w c o p r t c o p + w t o r q u e r t t o r q u e where w p , w v , w e , w r o o t , w c o m , w c o p , w t o r q u e are their respective weights. The pose reward r t p and velocity reward r t v match the current and task (target) motions in terms of the joint positions p t and velocities p ˙ t : r t p = exp [ − σ p ∑ j ‖ p ^ t j − p t j ‖ 2 ] (10) r t v = exp [ − σ v ∑ j ‖ p ˙ ^ t j − p ˙ t j ‖ 2 ] (11) where j is the index of joints, p ^ t j and p ˙ ^ t j are target position and velocity, respectively. The end-effector reward is defined to encourage the robot to track the target positions of selected end-effectors: r t e = exp [ − σ e ∑ i | | x ^ t i − x t i | | 2 ] (12) where i is the index of the end-effector. Let x t i be the position of end-effectors, that include left foot and right foot, relative to the moving coordinate frame attached to the root (waist structure). The end-effectors motions are supposed to match well if the joint angles match well, and vice versa.

We also design the root reward function r t r o o t to track the task root motion including the root’s position x ^ t r o o t and rotation q ^ t r o o t . r t r o o t = exp [ − σ r 1 ‖ x ^ t r o o t − x t r o o t ‖ 2 − σ r 2 ‖ q ^ t r o o t − q t r o o t ‖ 2 ] (13) The overall center of mass reward r t c o m is also considered in the learning process, enabling the control policy to track the target height during the complete squatting cycle. r t c o m = exp [ − σ c o m ‖ x ^ t c o m − x t c o m ‖ 2 ] (14)

The movement of system CoP is an important indicator of system balance as set forth in the introduction. When the CoP leaves or is about to leave the foot support polygon, a possible toppling of the foot or loss of balance is imminent. During squatting (or walking), there are always one or two feet on the ground and the support polygon on the touching foot persists, and thus the CoP criterion is highly relevant for characterizing the tipping equilibrium of a bipedal robot (Sardain and Bessonnet, 2004). When the CoP point lies within the foot support polygon, it can be assumed that the biped robot can keep balance during squatting. Considering the importance of CoP, we incorporate the CoP positions in the state input as a feedback from the controller and also add a CoP reward function to encourage the current CoP ( c t c o p ) to stay inside a stable region around the center of the foot support. By considering the geometric of the foot in the lower extremity exoskeleton design, the stable region for foot CoP is defined as a rectangle area S around the foot geometric center whose width and length are set to 7 c m and 11 c m , respectively (narrower in the lateral direction than forward direction). And the CoP reward function is expressed as r t c o p = { exp [ − σ c o p ‖ D ( c t c o p , S ) ‖ 2 ] , if  c t c o p ∈ S 0 , if  c t c o p ∉ S (15) where D ( ⋅ , ⋅ ) is the Euclidean distance between CoP and the center of S. The goal of this CoP reward is to encourage the controller to predict an action that will improve the balance and robustness of the exoskeleton’s motion.

At last, we design the torque reward to reduce energy consumption or improve efficiency and to prevent damaging joint actuators during the deployment. r t t o r q u e = exp [ − σ t o r q u e ∑ i ‖ τ i ‖ 2 ] (16) where i is the index of joints.

Due to the model discrepancy between the physics simulation and the real-world environment, well-known as reality or sim-to-real gap (Yu et al., 2018), the trained control policy usually performs poorly in the real environment. In order to improve the robustness of the controller against model inaccuracy and bridge the sim-to-real gap, we need to develop a robust control policy capable of handling various environments with different dynamics characteristics. To this end, we adopt dynamics randomization (Sadeghi and Levine, 2016; Tobin et al., 2017) in our training strategy, in which dynamics parameters of the simulation environment are randomly sampled from an uniform distribution for each episode. The objective in Eq. 5 is then modified to maximize the expected reward across a distribution of dynamics characteristics ρ ( μ ) : π * = arg max π E μ ∼ ρ ( μ ) E τ ∼ p ( τ | π , μ ) [ ∑ t = 0 T − 1 γ t r t ] , (17) where μ represents the values of the dynamics parameters that are randomized during training. By training policies to adapt to variability in environment dynamics, the resulting policy will be more robust when transferred to the real world.

To demonstrate the effectiveness of our RL-based robust controller, we train the lower extremity exoskeleton to imitate a 4s reference squatting motion that is manually created based on human squatting motion. The reference squatting motion can provide guidance for motion mimicking but needs not to be generated precisely. The exoskeleton contains eight joint DoFs actuated with motors, all of which are controlled by the RL controller. On each leg, there are has four actuators, including one actuator for hip flexion/extension, one actuator for knee flexion/extension, two actuators for ankle dorsiflexion/plantarflexion and ankle inversion/eversion, respectively. The open source library DART (Lee et al., 2018) is utilized to simulate the exoskeleton dynamics. The GRFs are computed by a Dantzig LCP (linear complementary problem) solver (Baraff, 1994). We utilize PyTorch (Paszke et al., 2019) to implement the neural network and the PPO method for the learning process. The networks are initialized by the Xavier uniform method (Glorot and Bengio, 2010). We use a desktop computer with an Intel^® Xeon(R) CPU E5-1660 v3 at 3.00 GHz × 16 to generate samples in parallel during training. Totally about 20 million samples are collected in each simulation. The policy and value networks are updated at a learning rate of 10 − 4 , which is linearly decreased to 0 when 20 million samples are collected. PPO algorithm is robust in that hyperparameter initialization is a bit more forgiving and it can handle a wide variety of RL tasks. We do not deliberately tune hyperparameters and just use the common setting as in the literature (Schulman et al., 2017; Tan et al., 2018). Hyper-parameters settings for training using PPO are shown in Table 1.

To verify the robustness of the trained controller, we test the control policies in out-of-distribution simulated environments, where the dynamic parameters of the exoskeleton are sampled randomly from a larger range of values than those during training. Table 2 shows the dynamics parameters details of the exoskeleton and their range during training and testing. Note that the observation latency denotes the observation time delay in the real physical system due to sensor noise and time delay during information transfer. Considering the observation latency improves the reality of the simulations and further increases the difficulty of policy training. The simulation frequency (time step for the environment simulation) and control policy output frequency are set to 900 and 30Hz, respectively. According to the PD torque Eq. 8, the parameters about the proportional gain k p and differential gain k v are set to 900 and 40, respectively. The differential gain k v is chosen to be sufficiently high to prevent unwanted oscillation on the exoskeleton robot. From our experience, the control performance is robust against the variance of gains to a certain extent. For instance, increasing or decreasing the position gain k p to 1200 or 800 does not noticeably change the control performance. We have tried different sets of rewards weights (in Eq. 9) in a proper range, we found the control performance with different sets of weights are similar, and we choose the best one: w p = 0.8 , w c o p = 0.8 , w v = 0.1 , w e e = 0.7 , w c o m = 0.4 , w r o o t = 0.7 , w t o r q u e = 0.1 . The torque limit for each joint is set to 100 N m in the simulation.