Bridging Reinforcement Learning and Iterative Learning Control: Autonomous Motion Learning for Unknown, Nonlinear Dynamics

Abstract

This work addresses the problem of reference tracking in autonomously learning robots with unknown, nonlinear dynamics. Existing solutions require model information or extensive parameter tuning, and have rarely been validated in real-world experiments. We propose a learning control scheme that learns to approximate the unknown dynamics by a Gaussian Process (GP), which is used to optimize and apply a feedforward control input on each trial. Unlike existing approaches, the proposed method neither requires knowledge of the system states and their dynamics nor knowledge of an effective feedback control structure. All algorithm parameters are chosen automatically, i.e. the learning method works plug and play. The proposed method is validated in extensive simulations and real-world experiments. In contrast to most existing work, we study learning dynamics for more than one motion task as well as the robustness of performance across a large range of learning parameters. The method’s plug and play applicability is demonstrated by experiments with a balancing robot, in which the proposed method rapidly learns to track the desired output. Due to its model-agnostic and plug and play properties, the proposed method is expected to have high potential for application to a large class of reference tracking problems in systems with unknown, nonlinear dynamics.

1 Introduction

Recent developments in robotic technology remarkably contribute to the quality of human live: Hazardous tasks on rescue missions are handled by mobile robots that rifle through wreckage to locate people in need of help (Murphy, 2004). Advances in medical robotics strive for minimizing complications during surgery (Coulson et al., 2008). And the combination of exoskeletons and control algorithms aims for a future in which people struck by disability can walk again (Harib et al., 2018). The way to such accomplishments is paved by control techniques that enable robots to precisely perform agile and dynamic motions.

For example, model predictive control can achieve accurate motion if a precise model of the dynamics is available (Apgar et al., 2018; Hehn and D’Andrea, 2012; Feng et al., 2014). Requirements regarding the model’s precision can be relaxed by robust or adaptive control techniques if the uncertainties comply with preset assumptions (Dong and Kuhnert, 2005; Dydek et al., 2013; Golovin and Palis, 2019). Under similar conditions, Iterative Learning Control (ILC) can overcome model uncertainties and unknown disturbances by learning from errors of previous trials (Muller et al., 2012; Seel et al., 2016). However, all of these control approaches require system-specific prior knowledge to craft a suited model, controller, or learning configuration. In contrast, autonomy requires a methodology that self-reliantly learns a solution to the control problem without requiring any system-specific prior knowledge. In particular, Reinforcement Learning (RL) techniques have been employed to solve complex motion tasks without requiring any prior information. However, RL solutions typically suffer from two major drawbacks: First, the vast majority of the results were obtained in simulated environments (Heess et al., 2017; Tassa et al., 2018; Tsounis et al., 2020). Second, the few results obtained in real-world environments required at least multiple hours of learning, and the resulting controllers can be prone to failure (Schuitema, 2012; Kalashnikov et al., 2018; Ha et al., 2020; Zeng et al., 2020). The only exception from this statement is given by RL methods that exploit task-specific knowledge in the form of good initial policies and require 60–300 trials for local policy optimization (Kober and Peters, 2008; Peters and Schaal, 2008; Kormushev et al., 2013). A major breakthrough with respect to robustness and data-efficiency was achieved by hybrid techniques that learn parameter-free models, namely Gaussian Processes (GP), but also employ system-specific information such as knowledge of a state vector and an effective state feedback structure (Deisenroth, 2010). In the prominent example of PILCO (Deisenroth and Rasmussen, 2011a), experimental data are used to approximate the unknown dynamics by a GP, which is used to determine the optimal parameters of a state feedback controller. By this approach, an inverted pendulum on a cart could be swung up and stabilized in 12 s of system interaction (Deisenroth and Rasmussen, 2011a). However, in the context of autonomous motion learning, GP-based learning methods still suffer from two drawbacks: First, previously proposed methods only solve set-point stabilization tasks, which do not enable robots to perform challenging, dynamic maneuvers. These require reference tracking. Second, GP-based learning techniques still require system-specific prior knowledge such as the configuration of cost functions, a state vector that fully describes the system dynamics, and a control structure that is effective with respect to the problem at hand. Hence, the methods are not suited for plug and play learning of highly dynamic robotic motions.

The present contribution proposes a GP-based learning method for autonomously solving highly dynamic reference tracking tasks in systems with unknown, nonlinear, single-input/single-output dynamics. The proposed method autonomously determines all of its necessary parameters such that plug and play application becomes feasible. The method’s capability to rapidly learn solutions to various reference tracking tasks while not requiring any system-specific prior knowledge is validated by extensive simulations and real-world experiments using a two-wheeled inverted pendulum robot, see Figure 1.

FIGURE 1

1.1 Related Work

Learning for control has been considered in a large body of literature that can be categorized by 1) the considered control problem respectively control strategy, 2) necessary system-specific prior knowledge, and 3) speed of learning. Reinforcement Learning (RL) techniques typically do not require any model and only few learning parameters such as step sizes or weights in cost functions. Furthermore, general RL approaches such as genetic algorithms (Moriarty and Miikkulainen, 2007) or policy gradient approaches (Peters and Schaal, 2006) can be applied to arbitrary control problems with unknown, nonlinear dynamics, but in turn require comparatively long periods of learning (Deisenroth and Rasmussen, 2011a). The speed of learning can be significantly increased if the technique is targeted towards a specific control problem and strategy such as stabilization by state feedback control, see e.g. (Lewis and Vrabie, 2009; Lewis and Vamvoudakis, 2011). A particularly data-efficient approach are so called model-based techniques that model the unknown, nonlinear dynamics by a GP, which is then used to design a state feedback controller (Deisenroth, 2011). Some successful applications to real-world examples are the control of a single inverted pendulum (Deisenroth and Rasmussen, 2011a), double inverted pendulum (Hesse et al., 2018), and robotic manipulator (Deisenroth et al., 2012). The concept of GP-based learning control has been further investigated in a variety of contributions. Stability of feedback-controlled GPs has been analyzed (Vinogradska et al., 2017, 2016), the problem of computational and data requirements has been investigated (Nguyen-Tuong and Peters, 2008; Capone et al., 2020), and solutions for safely improving an existing feedback controller have been proposed (Berkenkamp and Schoellig, 2015; Hewing et al., 2017; Umlauft et al., 2020). In a similar fashion, a type of lazy learning methods constructs locally weighted models based on experimental data to design feedback controllers for set-point stabilization, see (Atkeson et al., 1997), but require problem-specific knowledge like, e.g., the configuration of cost functions. While all of these works consider the challenging problem of efficiently learning control solutions for unknown, respectively uncertain, nonlinear dynamics, they have focused on the problem of set-point stabilization of systems, for which an effective feedback control structure is known. If the control tasks consists in performing a highly-dynamic motion, the achievable performance of time-domain feedback control is inherently limited by phenomena such as unknown delays, measurement noise, or non-minimum phase dynamics. To overcome the performance limitations of feedback control, a feedforward control component is required, see Figure 2.

FIGURE 2

In contrast to GP-based learning techniques, Iterative Learning Control (ILC) has focused on reference tracking tasks solved by feedforward control (Ahn et al., 2007; Arimoto et al., 1984). Model-based techniques like norm-optimal or ILC automatically determine the learning parameters, but require a model of the linear plant dynamics (Tayebi and Zaremba, 2002; Gunnarsson and Norrlöf, 2001; Amann et al., 1996). Model-free approaches like PD-ILC do not require a model but learning parameters that are typically tuned in experiment (Bristow et al., 2005). The concepts of PD-type (Shen et al., 2016) and norm-optimal (Lu et al., 2018) ILC have been extended to the case of nonlinear dynamics, but assume the dynamics to be known. To relax requirements with respect to available model information, recent research has focused on so called data-driven ILC (DD-ILC) (Hou and Wang, 2013), which does not require a model of the plant. In the case of nonlinear, unknown dynamics, DD-ILC methods typically employ dynamic linearization of the plant dynamics and estimate the gradient of said linearization (Chi et al., 2015a,b; Ai et al., 2020). Alternatively, neural networks (NN) have been employed in DD-ILC to model the unknown dynamics (Ma et al., 2020; Yu et al., 2020). In a similar fashion, (Petric et al., 2018) propose basis functions for computing the input trajectory to track a desired reference, whereby the weights of the basis functions are learned by weighted linear regression, but the learning requires multiple hand-tuned parameters including the number of basis functions, step-sizes, and damping constants. In summary, while existing DD-ILC and similar methods can solve reference tracking tasks without requiring a plant model, some system-specific prior knowledge is required as, e.g., the signs of the dynamic linearization (Chi et al., 2015a,b; Ai et al., 2020), the layout of a suited neural network (Ma et al., 2020; Yu et al., 2020), or weights and step-sizes in update laws (Petric et al., 2018).

In summary, we conclude that reference tracking tasks in systems with unknown, nonlinear dynamics can be solved by DD-ILC methods, which, however, require system-specific prior knowledge such that autonomous plug and play application is generally not possible. In contrast, set-point stabilization problems can be solved by GP-based learning methods that assume comparatively little system-specific prior knowledge. However, in the context of reference tracking tasks, GP-based learning methods suffer from the inherent limitations of feedback control. To the best of our knowledge, there exists no learning method that autonomously solves reference tracking tasks for unknown, nonlinear systems, employs feedforward control to overcome the limitations of feedback control, and does not require system-specific prior knowledge such that autonomous plug and play application is enabled.

1.2 Contributions

In this contribution, a GP-based ILC scheme is proposed that autonomously solves reference tracking tasks in systems with unknown, nonlinear, single-input/single-output dynamics. The proposed method includes a procedure to autonomously determine necessary parameters and enable plug and play application. Since the method directly models the input/output dynamics, only the output variable, instead of an entire state vector, has to be known and measured. To overcome the inherent limitations of feedback control, the proposed method employs feedforward control.

The proposed method is first validated by extensive simulations of a two-wheeled inverted pendulum robot (TWIPR), in which precise tracking is achieved after a small number of trials. Unlike existing approaches, the proposed method is not only verified for a single, well-chosen parameter configuration but for a wide range of parameter combinations such that robustness with respect to the autonomously determined parameters is ensured. In contrast to a variety of contributions, in which validation was restricted to simulated environments, the proposed method’s capability of solving real-world reference tracking tasks in a plug and play manor is validated by experiments on a TWIPR, see Figure 1.

2 Problem Formulation

Consider an autonomous system that can repeatedly attempt a reference tracking task, as, e.g., a robot trying to perform a desired maneuver. We assume that the system’s output, e.g., a joint angle or position, can be influenced by an input signal, e.g., a motor torque, and that the relation of these variables is deterministic, causal, and time-invariant. However, we do not assume that a model of the dynamics is available and we do assume the general case of nonlinear dynamics.

Formally, consider a discrete-time, single-input, single-output, repetitive system with a finite trial duration of samples, and, on trial and sample n ∈ [1, N], output variable , respectively input variable . The samples are collected in the so called output trajectory , respectively input trajectory , i.e., ,Without loss of generality, the dynamics can be written in the lifted formwhere p is the unknown, trial-invariant, nonlinear dynamics. The task consists of updating the input u_j from trial to trial such that the output y_j converges to the desired reference trajectory . Tracking performance is measured by the error trajectoryand root-mean-squared error (RMSE)

The problem considered in this work consists in developing a learning method that updates the input trajectory on each trial such that the RMSE decreases. Learning performance is judged based on the progression of the RMSE through trials, and the RMSE shall decline quickly and monotonically. The learning method must not require any a priori model information on the plant dynamics. To support plug and play application, the method must autonomously determine necessary parameters. Furthermore, the method must provide a fair degree of robustness with respect to autonomously determined parameters.

3 Proposed Learning Method

We address the proposed problem by an iterative learning scheme, in which each iteration consists of three steps. First, a parameter-free model of the plant dynamics is identified using the experimental data of previous trials, see Section 3.1. To accommodate for possibly nonlinear dynamics, a generic GP model is employed, which predicts the output trajectory for a given input trajectory. Second, the updated input trajectory is determined by solving an optimal feedfoward control problem based on the GP model, see Section 3.2. Third, the updated input trajectory is applied to the plant and resulting data is in turn used to refine the GP model. The structure of the proposed learning scheme is depicted in Figure 3. To enable plug and play application, the proposed method autonomously determines necessary parameters, see Section 3.3.

FIGURE 3

3.1 Gaussian Process Model

We propose a Gaussian Process (GP) model, formally a function , that predicts the plant’s output trajectory based on an input trajectory , where the trial index is omitted for sake of notational simplicity.

Let denote the unknown target function that depends on the regression vector . Predictions are based on observations stemming from:The K observation pairs (z_k, v_k) are collected in the observation training vector and regression training matrix , i.e.,

The kernel function of two regression vectors and is denoted by . The kernel matrix of two regression matrices , , which are assembled according to (8), is denoted by and has entries .

Given test regression vectors assembled in the regression matrix , the predicted mean and covariance are given by:

The general GP framework can be employed in different ways to model the unknown dynamics (Eq. 3), where the model characteristics are determined by the definition of observation variable z, regression vector v, and kernel function k. First, we exploit the dynamics’ time-invariance by employing a single GP for predicting each output sample. Hence, the observation variable and regression vector are time dependent, i.e., ∀n ∈ [1, N], z_n, v_n. Now, the model can be chosen to be one of three types, namely finite impulse response (FIR), infinite impulse response (IIR), or state space (SS), which are outlined in the following. A FIR model is obtained when the regression vector consists of the current and all previous input samples, i.e.,An IIR model is obtained when the regression vector consists of the current input and the previous output samples, i.e.,A SS model is obtained when the regression vector consists of the current input and the previous state sample, which is denoted by , i.e.,The SS model requires multiple GPs with each predicting the progression of a single state variable (Deisenroth and Rasmussen, 2011b), which not only increases computational complexity, but also requires measurements of the full state vector. Furthermore, IIR and SS models require so called roll-out predictions, meaning that the predictions of previous samples are required for predicting the current sample (Deisenroth and Rasmussen, 2011b), and, hence, the matrix has to be recomputed for each sample in the output trajectory, which increases the model’s complexity and computational demands. In contrast, the FIR model only requires a single prediction according to Eq. 9 and only a single computation of the matrix . We, hence, employ a FIR model, and the regression vector is defined according to Eq. 11 in order to reduce the computational demands of the learning method.

We further choose difference-predictions, i.e.,which, compared to absolute predictions, increase the model’s capability of extrapolation, see (Deisenroth and Rasmussen, 2011b).

To predict an output trajectory for an arbitrary input trajectory u, the latter is used to determine N regression vectors v_n, n ∈ [1, N], according to Eq. 11, which are assembled in a regression matrix V according to Eq. 8. The predicted mean vector μ follows from Eq. 9. By Eq. 14, μ contains difference predictions such that the components of follow from the cumulative sum of μ, i.e.

GP predictions are known to become computational expensive with increasing amounts of training data (Snelson and Ghahramani, 2006). To overcome this limitation, various data selection approaches have been proposed to reduce training data to a tractable amount, see, e.g., (Seeger et al., 2003; Snelson and Ghahramani, 2006). In the present work, we simply propose limiting the training data to the last trials.

3.2 Optimal Feedforward Control

3.3 Autonomous Parameterization

Ideally, autonomous learning methods should require neither a priori model information nor manual tuning of parameters. In contrary to previous contributions, the proposed method automatically determines necessary parameters by the procedure outlined in this section, and, as a result, plug and play application is enabled, see Figure 4.

FIGURE 4

First, we consider the choice of the initial data set that is used to determine the first GP model and consists of trajectory pairs (y_i, u_i), i.e.,For this purpose, we first determine the largest significant frequency f_O of the reference trajectory. The frequency f_O is used to design a zero-phase low-pass filter f_LP. The low-pass filter f_LP is applied to a zero mean normal distribution with covariance , and the initial input trajectories are drawn from the resulting distribution, i.e.,The input variance is iteratively increased until an input trajectory drawn according to Eq. 24 leads to an output trajectory, whose maximum roughly equals the maximum of the reference, i.e.,In autonomous parameterization, the number of initial trials is chosen as one, i.e., I = 1, to decrease the number of total trials required for the learning. Note that a larger number of initial trials reduces the variance of the convergence speed, i.e., in safety-critical applications a larger number of initial trials may be recommendable. However, using only one initial trial, the proposed method already provides a remarkably safe convergence, as will be demonstrated in Section 4.3. Hence, the autonomous parameterization employs I = 1.

Once the parameters f_O, , and I have been determined, the initial trials are performed, and the weights q and s are chosen based on the experimental data. The scalar q, which weights the error trajectory, is without loss of generality chosen as unity, i.e.,The scalar s, which weighs the change in input variable, is chosen as the average squared ratio of output to input maxima over the I initial trials, i.e.,The purpose of the weight selection in Eqs 26–27 is to normalize the cost function (Eq. 21), i.e., we would like the weighted change in the input trajectory to have an impact on the cost function that is equal to impact to the weighted next-trial error trajectory. To achieve this normalization, we employ the squared ratio of the initial trials’ input and output trajectories as described in Eq. 27. Note that the normalization of the cost function only depends on the ratio of the weights q and s but does not depend on their absolute values, hence the choice q = 1 is arbitrary and without loss of generality.

4 Validation by Simulation

In this section, the proposed learning method is validated by simulation of a two-wheeled inverted pendulum robot (TWIPR) that is meant to perform challenging maneuvers, see Figure 5. The TWIPR and automatic determination of learning parameters are presented in Section 4.1. Afterwards, the learning performance for three representative references is investigated in Section 4.2, and the proposed method’s robustness with respect to learning parameters is verified in Section 4.3. Lastly, the effect of the weight s on the learning characteristics is studied in Section 4.4.

FIGURE 5

4.1 The Learning Problem

Consider the TWIPR and three desired maneuvers depicted in Figure 5. The corresponding pitch angle reference trajectories are denoted by , , and and formal definitions are given in Supplementary Appendix S1.2. The robot consists of a pendulum body housing main electronics including a microcomputer, inertial measurement units, motors and accumulator. Wheels are mounted onto the motors such that the robot can drive while balancing its chassis. In order to validate the proposed method via simulations, a detailed, nonlinear model of the TWIPR dynamics, see Kim and Kwon, 2015, is implemented. However, the simulation model is completely unknown to the learning method, which can only interact with the simulation by applying an input trajectory and receiving the corresponding output trajectory. Only an approximate, linear model of the dynamics at the upright equilibrium has been obtained, which merely suffices to design a stabilizing feedback controller, see Supplementary Appendix S1.3. Due to the imprecise model, the feedback controller can not track the references precisely, and we instead employ the proposed learning method.

Instead of a state vector, the learning method only requires knowledge of the output variable, which is given by the pitch angle, i.e.,The input variable is given by the motor torque, ∀n ∈ [1, N], .

Application of the proposed learning method requires learning parameters that are automatically determined by the procedure outlined in Section 3.3. We aim at tracking pitch trajectories with a maximum of approximately 75° and spectral content roughly below 5 Hz, i.e.,Based on the frequency f_O, a forward-backward, second order Butterworth filter f_LP is designed, which is used for drawing initial input trajectories, see Eq. 24. To determine the input variance , the method automatically applies test input trajectories with successively increasing amplitudes to the nonlinear system as described in Section 3.3. A coarse grid is chosen, and the algorithm obtains e_I = [75, 75, 71, 20, −280] and thus selects , see Eq. 25 and Figure 6. Note that an even larger and finer grid as well as more sophisticated selection methods could be used, but this simple approach is sufficient because the proposed algorithm exhibits great robustness with respect to the choice of the input variance, as detailed in Section 4.3.

FIGURE 6

Next, the weights s and q of the cost function are determined. According to Eq. 26, q = 1 is selected. To determine the weight s, one initial trial with the previously determined input variance is performed. As detailed in Figure 7, the value of s directly results from Eq. 27 and the experimental data, i.e.,

FIGURE 7

To demonstrate the data-efficiency of the proposed learning method, the training data are limited to the last five trials, i.e., H = 5.

4.2 Learning Performance

First, learning performance for the desired references r₁, r₂, and r₃ is investigated. The parameters are chosen according to the previous section and only one initial trial I = 1 is used. In Figure 8, progressions of the output trajectories and error norms over the trials are depicted. For all three references, the proposed method achieves precise tracking after roughly 15 trials. The respective RMSEs rapidly decline over the first trials and converge to a small value close to zero. In case of the references r₂ and r₃, the RMSE decreases monotonically. The simulations demonstrate that, by using the automatically determined parameters, the proposed method rapidly learns to track three different reference trajectories without requiring any system-specific prior knowledge.

FIGURE 8

Lastly, Figure 8 also depicts the performance of a pure reinforcement learning method, namely a policy-gradient scheme, applied to the same learning task to serve as a baseline for comparison. Details of the policy-gradient algorithm are presented in Supplementary Appendix S1.4 and (Peters and Schaal, 2006). Here, we see that the RMSE declines at a rate that is by magnitudes slower compared to the learning speed of the proposed method. These results are of little surprise because the policy-gradient algorithm is a generic scheme that is not tailored towards the specific task of learning an input trajectory to track a desired reference trajectory. In contrast, the proposed approach leverages the fact that in reference tracking tasks the input/output dynamics of a nonlinear system can be effectively modelled by a FIR GP-model.

4.3 Robustness Analysis

The previous simulations have validated the method’s capability of achieving satisfying tracking performance when using the automatically determined parameters. However, as discussed above, presenting results for a single parameterization is of little value. Instead, a learning control method should, ideally, not only achieve satisfying performance for a single parameter configuration, but for a wide parameter space. This is a crucial prerequisite for a method that performs well on different systems for different reference trajectories without any manual adjustments.

To address this question, the proposed method’s robustness with respect to the automatically determined parameters is validated in the following study, where we aim at tracking reference r₁. We consider two different scenarios, namely, the greedy case of one initial trial, I = 1, and the conservative case of five initial trials, I = 5. Recall the two parameters s and , which are the weight in the optimal control problem and the initial input variance.

In the case of I = 1, the weight s is chosen from the set that consists of ten logarithmically spaced values and the initial input variance is chosen from the set that consists of ten quadratically spaced values, i.e., for I = 1.For each of the 100 parameter pairs in , 50 runs are performed. A runr consists of choosing a parameter pair from , producing I initial input trajectories, and executing the proposed learning method for an additional 50 trials such that a progression of the RMSE throughout trials is obtained, which we denote bywhere j ∈ [0, I + N] is the trial index, k ∈ [1, 100] is the parameter index, and r ∈ [1, 50] is the run index.

The same procedure is applied in the case of I = 5, but the parameters are chosen from larger sets, i.e., for I = 5.To evaluate performance, the maximum , 99th percentile , 75th percentile , and median of the RMSE over parameters and runs are considered. Formally,and , , are defined accordingly. Results depicted in Figure 9 show that, for both I = 1 and I = 5, the maximum of the RMSE converges to a value that is a roughly ten times smaller than the initial value such that the method’s robustness is validated. The RMSE’s 99^th percentile is monotonically decreasing, which implies that, besides single outliers, the method achieves the desired form of convergence as defined in Section 2. This means that the proposed method yields desirable performance for a large range of values of the weight s and the initial input variance . Furthermore, the RMSE’s median declines below a value of five degrees within 25 trials meaning that satisfying tracking performance is achieved. Lastly, it should be noted that a wider parameter space could be considered in the case of I = 5 meaning that, by increasing the amount of initial data, the robustness of the method can be further increased. We, hence, conclude that the simple approach proposed to autonomously determine the weight s and the input variance is more than sufficient and can be expected to suffice for more complex systems.

FIGURE 9

4.4 Effects of Weights

The previous analysis has shown that the method rapidly learns to track a desired reference while also being robust with respect to the automatically determined parameters. Next to the use case of automatic plug and play application, the method can also be tuned to meet the needs of a specific application. Hence, we next investigate how the choice of the weight s affects learning characteristics, namely the rate of convergence and robustness with respect to initial data. For this purpose, we consider the weightsThe remaining learning parameters are chosen as one initial trial I = 1 and an initial input variance . For each weight, 50 runs with differing initial data are performed and performance is judged based on the RMSE’s 90^th percentile and median over the 50 runs.

Results depicted in Figure 10 show that for a comparatively large value of s = 10⁰, the RMSE monotonically declines at a slow pace. Furthermore, there is hardly any difference between median and 90^th percentile performance. Decreasing the value to s = 10^–2 leads to a significant increase in speed of convergence. Speed of median convergence can be further increased by lowering the value of the weight to s = 10^–4, which, however, comes at the price of larger 90^th percentile RMSEs, which imply an increase in performance variance. If the weight is lowered to an even smaller value, s = 10^–6, median performance is not further increased, but the 90^th percentile RMSE does no longer converge meaning that learning fails in a significant portion of runs. In summary, the study indicates that the weight s may be used to tune learning behavior, whereby comparatively large values of s lead to slow learning that is robust with respect to initial data. Decreasing the value of s can increase the speed of learning, but may come at the price of sensitivity with respect to initial data.

FIGURE 10

5 Validation by Experiment

To demonstrate the plug and play applicability of the proposed learning method, it is applied to a real-world TWIPR, which has been previously used to validate learning control methods (Meindl et al., 2020). The robot is meant to dive beneath an obstacle as depicted in Figure 2 with the corresponding reference trajectory , ∀n ∈ [1, 75],First, the proposed method determines the learning parameters yielding I = 1, , and s = 0.1. The initial input trajectory is drawn according to (Eq. 24) and applied to the TWIPR. The corresponding output trajectory significantly differs from the reference with a RMSE of roughly 75°, see Figure 11. From here onwards, the method iteratively determines a GP model, updates the input trajectory, and performs an experimental trial. Once learning begins, the RMSE rapidly declines, the RMSE drops below 20° on the fourth trial, and a RMSE of less than 10° is reached on the eighth trial. Sufficiently precise tracking precision for diving beneath the obstacle is achieved on the seventh trial and a RMSE close to zero is achieved on the 10th trial. Note, that the RMSE slightly increases on some of the trials, which is likely due to the initial conditions varying from trial to trial.

FIGURE 11

In summary, the experiments validate that the proposed method enables a real-world robot with unknown, nonlinear dynamics to learn a challenging maneuver. Not only did learning require a small number of trials but the method could also be applied in a plug and play manor without iterative tuning of parameters.

6 Discussion and Conclusion

In this work, a GP-based learning control scheme has been proposed that autonomously solves reference tracking tasks in systems with unknown, nonlinear dynamics. On each iteration, the unknown dynamics are approximated by a Gaussian Process (GP), which is then used to determine and apply an optimal feedforward control input. The method is completely plug and play, since all required algorithms parameters are determined automatically and manual tuning is avoided. The effectiveness and efficiency of the method were demonstrated by simulations and experiments using the example of a two-wheeled inverted pendulum robot that rapidly learns to perform several challenging maneuvers without any manual tuning or system-specific prior knowledge.

In contrast to previous GP-based learning control approaches, the proposed method overcomes the inherent limitations of time-domain feedback control; it neither assumes knowledge of an effective feedback control structure, nor does it assume the entire state vector to be known. Instead, the proposed method directly adjusts the input based on the measured output. It is therefore as model-agnostic and independent of system-specific prior knowledge as pure reinforcement learning schemes.

While reinforcement learning approaches typically require hundreds of trials for convergence and are therefore unsuitable for experimental validation, the proposed learning control method solves reference tracking problems in a small two-digit number of trials and was successfully validated in real-world experiments.

While the vast majority of previous contributions either validate methods only in simulations or provide only a single results for one carefully chosen parameterization and one specific motion, the present validation has proven effectiveness of the proposed method for several different motions and a large range of algorithm parameterizations. We thereby demonstrated robustness with respect to the automatically determined parameters, and we further investigated the effect of the learning weights on the trade-off between speed of learning and robustness.

We believe that the proposed method is highly suitable for use in kinematic systems that must perform challenging, highly dynamic maneuvers. Beyond the use case of rigid robotics, we expect the proposed method to have a major impact on the development of soft robotics, exoskeletons, and neuroprosthetics, and will therefore contribute to the evolution of autonomous robotic systems that rapidly learn to perform complex, dynamic motions under unknown conditions.

Despite these achievements, the proposed method is subject to remaining limitations. First, only learning to track a single reference trajectory was considered, and future research is going to extend the method to multi-reference tracking tasks. This can either be achieved by the concept outlined in Remark 3 or by combining the proposed method with model-free feeback controller learning to handle trial-varying references and disturbances. Second, the proposed method requires a feasible reference trajectory which might not be directly available in some applications. If the reference is only specified at a subset of the trial’s samples, the proposed method might be extended by well-established P2P-ILC concepts, see (Freeman and Tan, 2013; Janssens et al., 2013; Huo et al., 2020). And in cases in which the motion task is formulated only via goal states and constraints, a prior planning step might be required. Third, future research is going to extend the method to be applicable to multi-input/multi-output systems by implementing multiple GP models with each predicting the progression of one of the respective output variables. This may require adjustments like, e.g., sparse GP models, see (Rasmussen and Williams, 2005), in order to reduce the computational demands to a tractable amount. Fourth, the proposed method was validated on one real-world application, and future research will be concerned with applying the method to other, complex real-world applications.

References

• 1

  AhnH.-S.ChenY.MooreK. L. (2007). Iterative Learning Control: Brief Survey and Categorization. IEEE Trans. Syst. Man. Cybern. C37, 1099–1121. 10.1109/tsmcc.2007.905759

  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  AiQ.KeD.ZuoJ.MengW.LiuQ.ZhangZ.et al (2020). High-Order Model-free Adaptive Iterative Learning Control of Pneumatic Artificial Muscle with Enhanced Convergence. IEEE Trans. Ind. Electron.67, 9548–9559. 10.1109/TIE.2019.2952810

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  AmannN.OwensD. H.RogersE. (1996). Iterative Learning Control for Discrete-Time Systems with Exponential Rate of Convergence. IEE Proc. Control Theory Appl.143, 217–224. 10.1049/ip-cta:19960244

  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  ApgarT.ClaryP.GreenK.FernA.HurstJ. (2018). “Fast Online Trajectory Optimization for the Bipedal Robot Cassie,” in Robotics: Science And Systems XIV (Robotics: Science and Systems Foundation). 10.15607/rss.2018.xiv.054

  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  ArimotoS.KawamuraS.MiyazakiF. (1984). Bettering Operation of Robots by Learning. J. Robot. Syst.1, 123–140. 10.1002/rob.4620010203

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  AtkesonC. G.MooreA. W.SchaalS. (1997). Lo Cally Weighted Learning for Control. Artif. Intell. Rev.11, 75–113. 10.1023/a:1006511328852

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  BerkenkampF.SchoelligA. P. (2015). Safe and Robust Learning Control with Gaussian Processes. 2015 Eur. Control Conf. ECC2015, 2496–2501. 10.1109/ECC.2015.7330913

  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  BristowD. A.TharayilM.AlleyneA. G.HanZ. Z. (2005). A Survey of Iterative Learning Control. Kongzhi yu Juece/Control Decis.20, 961–966. 10.1109/mcs.2006.1636313

  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  CaponeA.NoskeG.UmlauftJ.BeckersT.LedererA.HircheS. (2020). Localized Active Learning of Gaussian Process State Space Models. arXiv 120, 1–10.

  • Google Scholar
  • View reference in article

• 10

  ChiR.HouZ.HuangB.JinS. (2015a). A Unified Data-Driven Design Framework of Optimality-Based Generalized Iterative Learning Control. Comput. Chem. Eng.77, 10–23. 10.1016/j.compchemeng.2015.03.003

  • CrossRef
  • Google Scholar
  • View reference in article

• 11

  ChiR.HouZ.HuangB.JinS. (2015b). A Unified Data-Driven Design Framework of Optimality-Based Generalized Iterative Learning Control. Comput. Chem. Eng.77, 10–23. 10.1016/j.compchemeng.2015.03.003

  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  CoulsonC. J.TaylorR. P.ReidA. P.GriffithsM. V.ProopsD. W.BrettP. N. (2008). An Autonomous Surgical Robot for Drilling a Cochleostomy: Preliminary Porcine Trial. Clin. Otolaryngol.33, 343–347. 10.1111/j.1749-4486.2008.01703.x

  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  DeisenrothM. P. (2011). A Survey on Policy Search for Robotics. FNT Robotics2, 1–142. 10.1561/2300000021

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  DeisenrothM. P. (2010). Efficient Reinforcement Learning Using Gaussian Processes, 9. KIT Scientific Publishing.

  • Google Scholar
  • View reference in article

• 15

  DeisenrothM. P.RasmussenC. E. (2011a2011). PILCO: A Model-Based and Data-Efficient Approach to Policy Search. Proc. 28th Int. Conf. Mach. Learn. ICML, 465–472.

  • Google Scholar
  • View reference in article

• 16

  DeisenrothM. P.RasmussenC. E. (2011b2011). PILCO: A Model-Based and Data-Efficient Approach to Policy Search. Proc. 28th Int. Conf. Mach. Learn. ICML, 465–472.

  • Google Scholar
  • View reference in article

• 17

  DeisenrothM.RasmussenC.FoxD. (2012). Learning to Control a Low-Cost Manipulator Using Data-Efficient Reinforcement Learning. Robotics Sci. Syst.7, 57–64. 10.15607/rss.2011.vii.008

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  Duy Nguyen-TuongD.PetersJ. (2008). “Local Gaussian Process Regression for Real-Time Model-Based Robot Control,” in 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems (Manhatten, New York: IEEE), 380–385. 10.1109/IROS.2008.4650850

  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  DydekZ. T.AnnaswamyA. M.LavretskyE. (2013). Adaptive Control of Quadrotor UAVs: A Design Trade Study with Flight Evaluations. IEEE Trans. Contr. Syst. Technol.21, 1400–1406. 10.1109/tcst.2012.2200104

  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  FengS.WhitmanE.XinjilefuX.AtkesonC. G. (2014). “Optimization Based Full Body Control for the Atlas Robot,” in 2014 IEEE-RAS International Conference on Humanoid Robots (IEEE). 10.1109/humanoids.2014.7041347Optimization Based Full Body Control for the Atlas Robot

  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  FreemanC. T.TanY. (2013). Iterative Learning Control with Mixed Constraints for Point-to-point Tracking. IEEE Trans. Contr. Syst. Technol.21, 604–616. 10.1109/TCST.2012.2187787

  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  GolovinI.PalisS. (2019). Robust Control for Active Damping of Elastic Gantry Crane Vibrations. Mech. Syst. Signal Process.121, 264–278. 10.1016/j.ymssp.2018.11.005

  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  GunnarssonS.NorrlöfM. (2001). On the Design of ILC Algorithms Using Optimization. Automatica37, 2011–2016. 10.1016/S0005-1098(01)00154-6

  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  HaS.XuP.TanZ.LevineS.TanJ. (2020). Learning to Walk in the Real World with Minimal Human Effort. arXiv preprint arXiv:2002.08550.

  • Google Scholar
  • View reference in article

• 25

  HaribO.HereidA.AgrawalA.GurrietT.FinetS.BoerisG.et al (2018). Feedback Control of an Exoskeleton for Paraplegics: Toward Robustly Stable, Hands-free Dynamic Walking. IEEE Control Syst.38, 61–87. 10.1109/mcs.2018.2866604

  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  HeessN.TbD.SriramS.LemmonJ.MerelJ.WayneG.et al (2017). Emergence of Locomotion Behaviours in Rich Environments. arXiv preprint arXiv:1707.02286.

  • Google Scholar
  • View reference in article

• 27

  HehnM.D’AndreaR. (2012). “Real-time Trajectory Generation for Interception Maneuvers with Quadrocopters,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 4979–4984. 10.1109/iros.2012.6386093

  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  HesseM.TimmermannJ.HüllermeierE.TrächtlerA. (2018). A Reinforcement Learning Strategy for the Swing-Up of the Double Pendulum on a Cart. Procedia Manuf.24, 15–20. 10.1016/j.promfg.2018.06.004

  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  HewingL.KabzanJ.ZeilingerM. N. (2017). Cautious Model Predictive Control Using Gaussian Process Regression. arXiv 28, 2736–2743.

  • Google Scholar
  • View reference in article

• 30

  HouZ.-S.WangZ. (2013). From Model-Based Control to Data-Driven Control: Survey, Classification and Perspective. Inf. Sci.235, 3–35. 10.1016/j.ins.2012.07.014

  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  HuoB.FreemanC. T.LiuY. (2020). Data-driven Gradient-Based Point-to-point Iterative Learning Control for Nonlinear Systems. Nonlinear Dyn.102, 269–283. 10.1007/s11071-020-05941-8

  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  JanssensP.PipeleersG.SweversJ. (2013). A Data-Driven Constrained Norm-Optimal Iterative Learning Control Framework for Lti Systems. IEEE Trans. Contr. Syst. Technol.21, 546–551. 10.1109/TCST.2012.2185699

  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  KalashnikovD.IrpanA.PastorP.IbarzJ.HerzogA.JangE.et al (2018). Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (PMLR). Proc. Mach. Learn. Res.87, 651–673.

  • Google Scholar
  • View reference in article

• 34

  KimS.KwonS. (2015). Dynamic Modeling of a Two-Wheeled Inverted Pendulum Balancing Mobile Robot. Int. J. Control Autom. Syst.13, 926–933. 10.1007/s12555-014-0564-8

  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  KoberJ.PetersJ. (2008). “Policy Search for Motor Primitives in Robotics,” in Advances in Neural Information Processing Systems. Editors KollerD.SchuurmansD.BengioY.BottouL. (Red Hook: Curran Associates, Inc.), 21.

  • Google Scholar
  • View reference in article

• 36

  KormushevP.CalinonS.CaldwellD. (2013). Reinforcement Learning in Robotics: Applications and Real-World Challenges. Robotics2, 122–148. 10.3390/robotics2030122

  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  LewisF. L.VamvoudakisK. G. (2011). Reinforcement Learning for Partially Observable Dynamic Processes: Adaptive Dynamic Programming Using Measured Output Data. IEEE Trans. Syst. Man. Cybern. B41, 14–25. 10.1109/TSMCB.2010.2043839

  • CrossRef
  • Google Scholar
  • View reference in article

• 38

  LewisF. L.VrabieD. (20092009). “Adaptive Dynamic Programming for Feedback Control,” in Proceedings of 2009 7th Asian Control Conference (Manhatten, New York: IEEE), 1402–1409.

  • Google Scholar
  • View reference in article

• 39

  LewisF. L.VrabieD.SyrmosV. L. (2012). Optimal Control. John Wiley & Sons.

  • Google Scholar
  • View reference in article

• 40

  LuJ.CaoZ.ZhangR.GaoF. (2018). Nonlinear Monotonically Convergent Iterative Learning Control for Batch Processes. IEEE Trans. Ind. Electron.65, 5826–5836. 10.1109/TIE.2017.2782201

  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  MaL.LiuX.KongX.LeeK. Y. (2021). Iterative Learning Model Predictive Control Based on Iterative Data-Driven Modeling. IEEE Trans. Neural Netw. Learn. Syst.32, 3377–3390. 10.1109/TNNLS.2020.3016295

  • CrossRef
  • Google Scholar

• 42

  MeindlM.MolinariF.RaischJ.SeelT. (2020). Overcoming Output Constraints in Iterative Learning Control Systems by Reference Adaptation. arXiv preprint arXiv:2002.00662.

  • Google Scholar

• 43

  MoriartyD. E.MiikkulainenR. (2007). Efficient Reinforcement Learning through Symbiotic Evolution, 11, 32. 10.1007/978-0-585-33656-5_3

  • CrossRef
  • Google Scholar

• 44

  MullerF. L.SchoelligA.D’AndreaR. (2012). “Iterative Learning of Feed-Forward Corrections for High-Performance Tracking,” in Proc. Of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (Manhatten, New York: IEEE), 3276–3281.

  • Google Scholar

• 45

  MurphyR. R. (2004). Activities of the Rescue Robots at the World Trade Center from 11-21 September 2001. IEEE Robot. Autom. Mag.11, 50–61. 10.1109/mra.2004.1337826

  • CrossRef
  • Google Scholar

• 46

  PetersJ.SchaalS. (20062006). Policy Gradient Methods for Robotics. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE). 10.1109/iros.2006.282564

  • CrossRef
  • Google Scholar

• 47

  PetersJ.SchaalS. (2008). Reinforcement Learning of Motor Skills with Policy Gradients. Neural Netw. Neurosci.21, 682–697. 10.1016/j.neunet.2008.02.003

  • CrossRef
  • Google Scholar

• 48

  PetricT.GamsA.ColasantoL.IjspeertA. J.UdeA. (2018). Accelerated Sensorimotor Learning of Compliant Movement Primitives. IEEE Trans. Robot.34, 1636–1642. 10.1109/tro.2018.2861921

  • CrossRef
  • Google Scholar

• 49

  RasmussenC. E.WilliamsC. K. I. (2005). Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press.

  • Google Scholar

• 50

  SchuitemaE. (2012). Reinforcement Learning on Autonomous Humanoid Robots. Ph.D. thesis. Delft, Netherlands: Delft University of Technology. 10.4233/UUID:986EA1C5-9E30-4AAC-AB66-4F3B6B6CA002

  • CrossRef
  • Google Scholar

• 51

  SeegerM.WilliamsC. K. I.LawrenceN. D. (2003). “Fast Forward Selection to Speed up Sparse Gaussian Process Regression,” in Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics. Editors BishopC. M.FreyB. J. (Key West, FL.

  • Google Scholar

• 52

  SeelT.WernerC.RaischJ.SchauerT. (2016). Iterative Learning Control of a Drop Foot Neuroprosthesis - Generating Physiological Foot Motion in Paretic Gait by Automatic Feedback Control. Control Eng. Pract.48, 87–97. 10.1016/j.conengprac.2015.11.007

  • CrossRef
  • Google Scholar

• 53

  ShenD.ZhangW.XuJ.-X. (2016). Iterative Learning Control for Discrete Nonlinear Systems with Randomly Iteration Varying Lengths. Syst. Control Lett.96, 81–87. 10.1016/j.sysconle.2016.07.004

  • CrossRef
  • Google Scholar

• 54

  SnelsonE.GhahramaniZ. (2006). “Sparse Gaussian Processes Using Pseudo-inputs,” in Advances in Neural Information Processing Systems, 1257–1264.

  • Google Scholar

• 55

  TassaY.DoronY.MuldalA.ErezT.LiY.de Las CasasD.et al (2018). Deepmind Control suiteCoRR Abs/1801, 00690.

  • Google Scholar

• 56

  TayebiA.ZarembaM. B. (2002). Robust ILC Design Is Straightforward for Uncertain LTI Systems Satisfying the Robust Performance Condition. IFAC Proc. Vol.35, 445–450. 10.3182/20020721-6-es-1901.01060

  • CrossRef
  • Google Scholar

• 57

  TsounisV.AlgeM.LeeJ.FarshidianF.HutterM. (2020). Deepgait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning. IEEE Robot. Autom. Lett.5, 3699–3706. 10.1109/LRA.2020.2979660

  • CrossRef
  • Google Scholar

• 58

  UmlauftJ.BeckersT.CaponeA.LedererA.HircheS. (2020). Smart Forgetting for Safe Online Learning with Gaussian Processes. Learning for Dynamics & Control. Proceedings in Machine Learning Research (PMLR), 1–10.

  • Google Scholar

• 59

  VinogradskaJ.BischoB.Nguyen-TuongD.PetersJ. (2017). Stability of Controllers for Gaussian Process Dynamics. J. Mach. Learn. Res.18, 1–37.

  • Google Scholar

• 60

  VinogradskaJ.BischoffB.Nguyen-TuongD.SchmidtH.RomerA.PetersJ. (20162016). Stability of Controllers for Gaussian Process Forward Models. 33rd Int. Conf. Mach. Learn. ICML2, 819–828.

  • Google Scholar

• 61

  Wenjie DongW.KuhnertK.-D. (2005). Robust Adaptive Control of Nonholonomic Mobile Robot with Parameter and Nonparameter Uncertainties. IEEE Trans. Robot.21, 261–266. 10.1109/TRO.2004.837236

  • CrossRef
  • Google Scholar

• 62

  YuQ.HouZ.BuX.YuQ. (2020). RBFNN-based Data-Driven Predictive Iterative Learning Control for Nonaffine Nonlinear Systems. IEEE Trans. Neural Netw. Learn. Syst.31, 1170–1182. 10.1109/tnnls.2019.2919441

  • CrossRef
  • Google Scholar

• 63

  ZengA.SongS.LeeJ.RodriguezA.FunkhouserT. (2020). Tossingbot: Learning to Throw Arbitrary Objects with Residual Physics. IEEE Trans. Robot.36, 1307–1319. 10.1109/TRO.2020.2988642

  • CrossRef
  • Google Scholar