Kick Control: Using the Attracting States Arising Within the Sensorimotor Loop of Self-Organized Robots as Motor Primitives

1. Introduction

The sensorimotor system is in general a product of evolution, development, learning, and adaptation (Todorov, 2004). One may examine alternatively whether self-organizing principles (Prokopenko et al., 2009) are capable to generate locomotion, in particular for the case of embodied (Ghazi-Zahedi et al., 2017) and/or biologically inspired robots (Pfeifer et al., 2007). Self-organization may serve in this context to generate a palette of behavioral primitives (Tani and Ito, 2003), or, on a higher level, to generate complex and playful behavior (Martius et al., 2013).

Attracting states in the sensorimotor loops corresponding to regular and exploratory motion, that is respectively to limit cycles and chaotic attractors, can be generated following two complementary routes. Within the first approach, which is especially suited for settings involving a large number of degrees of freedom (Kubisch et al., 2011), the optimal mapping between sensors and actuators is learned. Learning is on the other side absent when generative principles are implemented and studied (Gros, 2014). Short-term synaptic plasticity, a transient form of mostly presynaptic neural plasticity (Hennig, 2013), has been shown in this context to generate limit cycles (Toutounji and Pasemann, 2014) and chaotic attractors (Martin et al., 2016). Other examples are the homeostatic principles regulating the average neural activity (Linkerhand and Gros, 2013), which have been shown to induce surprisingly complex locomotive patters (Sándor et al., 2015).

The control of wheeled robots, e.g., with skid steering (Kozłowski and Pazderski, 2004), is well established, with explicit mathematical models (Das et al., 2006) being often the basis for the regulation of either the angular velocity of the individual wheels (Jimenez-Fernandez et al., 2012), or of the respective torque (Mandow et al., 2007). One may obtain the sensorimotor mapping relevant for the neuromorphic robot at hand also by training large neural networks (Conradt et al., 2015). As an alternative, we consider here exceedingly simple neural control schemes that are based on physical principles and not on adaptive learning. Typically, we need just one or two neurons per actuator.

Our starting point is the observation that neural activity, such as the spiking rate y, has a defined but limited range, say y ∈ [0, 1]. The activity of an output motor neuron could therefore be mapped, in principle, directly to the target angular velocity ω of the wheel, e.g., via ω ~ (2y − 1). Forward and backward motions would correspond in this setting to distinct neural activity patterns. We examine here in contrast a neural controller for which the time reversal symmetry between forward- and backward motion is broken spontaneously under the influence of initial conditions.

Our robots are equipped with two active wheels and a third passive support wheel. A maximum of two neurons per wheel generate self-organized locomotion, which is both compliant and variable. A simulated transmission rod is used to map the forth-and-back motion of the neural activity level y ∈ [0, 1] to the motor command in a manner that mirrors the transmission mechanism used by traditional steam locomotives to transmit the force generated by the pressurized steam piston to the rotating wheel.

Both computer simulations and experiments with real robots are performed in order to assess the feasibility of the proposed control mechanism. We find that locomotion is generated robustly both for individual robots and for trains of passively coupled two-wheeled cars. The only sensory information driving the neural activity is propriosensoric, namely the current angle of the wheel the neuron controls. Cross-wheel information exchange is absent. Highly complex behavioral patterns (such as forward and backward locomotion, exploratory chaotic motion) emerge nevertheless upon interaction with the environment, which modifies the attractor landscapes of the individual wheels.

Experimenting with additional top-down control signals we find that it is possible to kick individual wheels from one attractor into the basin of attraction of another attracting state. The self-organized limit cycles and chaotic attractors forming in the sensorimotor loop may hence be used also as motor primitives.

The rest of the paper is structured as follows. In section 2, the concept of kick control is introduced and defined mathematically, followed by the description of the proposed controller and of the experimental setup. In section 3, three possible implementations of kick control and their reliability tests are presented as a proof of concept. The experimental findings are then investigated via a simple analytic model as well. Finally, a summary is given in section 5.

2. Materials and Methods

From a general perspective we are interested in attracting states that form in the sensorimotor loop. Defining with x_R and x_E the dynamical variables of the robot (R) and of the environment (E) we have

$\begin{array}{ll}{\stackrel{·}{\text{x}}}_R={\text{f}}_R({\text{x}}_R,{\text{x}}_E;{\text{P}}_R),{\stackrel{·}{\text{x}}}_E={\text{f}}_E({\text{x}}_R,{\text{x}}_E;{\text{P}}_E)& (1)\end{array}$

for the combined dynamics, where P_R and P_E parametrize respectively the time evolution of the robot and of the environment. Parameters distinguish themselves in our notation from variables in the respect that they change either only very slowly, as the result of a separation of time scales, or via actions that can be considered external. Control signals will modify the evolution equations (1), which describe as such locomotion generated autonomously within the sensorimotor loop.

2.1. Kick Control

Locomotion is characterized typically by timescales of seconds. One speaks of “kick control,” when a robot is subjected to control sequences that are shorter than the time needed to complete a movement, e.g., of the order of 50–200 ms. Kick control is functionally dependent on the existence of multiple attracting states in the sensorimotor loop that correspond to distinct locomotive patterns. The control signal then serves “to kick” the dynamical system (1) into the basin of attraction of the desired attracting state. There are two mutually not excluding venues.

– “Frozen” kick control is present when the control pulse Δx_R acts via

$\begin{array}{ll}{\text{x}}_R\to {\text{x}}_R+\Delta {\text{x}}_R& (2)\end{array}$

exclusively onto the variables x_R of the robot. The parameters P_R are not changed, they remain frozen. The state of the system is kicked here from its present state x_R to a new state, viz to ${\text{x}}_R^{\prime }={\text{x}}_R+\Delta {\text{x}}_R$. A sudden change of parameters as in (2) corresponds to an additional strong temporary force within the right-hand-side of the corresponding Newton equation of motion (i.e., to a kick).

– “Quenched” kick control is realized when the control pulse ΔP_R(t), which may be something like a rectangular pulse or a broadened δ-function, leads via

$\begin{array}{ll}{\text{P}}_R\to {\text{P}}_R+\Delta {\text{P}}_R(t)& (3)\end{array}$

to a sudden but transient change of the parameters P_R of the robot. The near instantaneous change of parameters catapults the system into a quenched configuration for which the attracting states of evolution equations (1) are morphed.

Quenched kick control will need in general a somewhat longer control signal. The reason is that the time evolution under the influence of the morphed attractors, that are present over the duration of the control pulse for the case of quenched kick control, needs to progress to a point in which the system finds itself within the basin of attraction of a different attractor once the control pulse ceases.

We have defined control signals as changes of either the variables or of the parameters of the robot. We may consider alternative events that lead to changes of the state of the environment, notably of the parameters P_E. This happens in particular when the robot interacts with other objects, e.g., when it bounces against a wall. The resulting transition to a different attracting state may hence also be described at times within the terminology of kick control.

2.2. Simulated Steam-Locomotive Actuator

We consider robots for which the active wheels are controlled independently by simple proprioceptual rate-encoding neurons. A finite angular velocity is attained when the internally generated torque interacts with the external response resulting from friction forces, gravity and inertia.

Neural activity covers a finite range, which can be normalized to the interval [0, 1]. A straightforward route for translating the neural activity y_i to a rotational mode would be to take y_i to be directly proportional, in the spirit of direct control, to a target angular velocity. Here we consider an alternative mechanism which allows for the generation of self-stabilizing attractors in the sensorimotor loop. For this purpose we use two steam-locomotive-like actuators that allow to translate the finite range of neural activity into a rotational model.

The two simulated actuators used for each wheel have a perpendicular alignment that allows for a continuous tracking. The respective transmission rods are fixed at one point of the perimeter of the wheel, as illustrated in Figure 1, being moved at the other end by ideal springs with constant k. The spring forces

$\begin{array}{ll}{F}_i=k\left(x_i^{\mathrm{\text{(t)}}}-x_i^{\mathrm{\text{(a)}}}\right),i=1,2,& (4)\end{array}$

are proportional to the distance $x_i^{\mathrm{\text{(t)}}}-x_i^{\mathrm{\text{(a)}}}$ between the normalized target position $x_i^{\mathrm{\text{(t)}}}$ and the actual position $x_i^{\mathrm{\text{(a)}}}$ of the wheel. The actual positions $x_i^{\mathrm{\text{(a)}}}\in \left[-1,1\right]$ are determined in turn by the projections of the rotational angle φ of the wheel, measured respectively relative the horizontal and the vertical direction:

$\begin{array}{ll}{x}_1^{\mathrm{\text{(a)}}}=cos\phi ,x_2^{\mathrm{\text{(a)}}}=sin\phi .& (5)\end{array}$

The target positions $x_i^{\mathrm{\text{(t)}}}\in (-1,1)$ are provided on the other side by the output of two independent rate-encoding neurons,

$\begin{array}{ll}{x}_i^{\mathrm{\text{(t)}}}=2y(x_i)-1,y(x)=\frac{1}{1+e^{-ax}},& (6)\end{array}$

which are characterized by a sigmoidal transfer function y(x) ∈ [0, 1] with slope a/4. The dynamics of the membrane potential x_i is driven in turn by the proprioceptual input $x_i^{\mathrm{\text{(a)}}}$,

$\begin{array}{ll}\tau {ẋ}_i=x_i^{\mathrm{\text{(a)}}}-x_i,i=1,2,& (7)\end{array}$

where the internal time scale τ is of the order of a few hundred milliseconds. The total tangential force acting on the wheel is then the vectorial sum of the projections of individual spring forces (see Figure 1):

$\begin{array}{ll}{F}^{\mathrm{\text{tan}}}=F_1^{\mathrm{\text{tan}}}+F_2^{\mathrm{\text{tan}}}=F_{1}sin\phi -F_{2}cos\phi .& (8)\end{array}$

FIGURE 1

Figure 1. Left: The two-wheeled real robot constructed with the LEGO Mindstorms package. The active wheels are controlled by independent motors. A third passive wheel (the spherical shaped metallic wheel below the robots) keeps the body horizontal. Right: A sketch of a wheel with two perpendicular actuators (click for movie). A spring with spring constant k pulls the rod (in gray) toward the target position $x_i^{(t)}$ (green). The target position $x_i^{(t)}$ is determined by the output of a controlling neuron, as described by Equation (6), which receives in turn the actual position $x_i^{(a)}$ (orange) of the wheel as an input. Compare Equation (5). The final torque acting on the wheel is given by the sum of the tangential components of the spring forces F₁ and F₂, see Equation (8).

The proposed controller translates hence the forth and back dynamics of the normalized neural activity y ∈ [0, 1] to a rotational motion for the wheel. For an animated illustration of the steam-locomotive controller see the Supplementary Material.

Note that a single actuator, e.g., a single horizontal transmission rod, would lead to a sinusoidal force $\propto \text{\hspace{0.17em}}{F}_1^{\text{tan}}$ vanishing at φ = 0 and φ = π. The combined force (8) is on the other hand always finite when two actuators with a perpendicular alignment are employed. See Figure 1 and the illustrating movie presented in the Supplementary Material.

2.3. LEGO Mindstorms Robot

To test the presented actuators we constructed robots with two active wheels using the LEGO Mindstorms Education Core Set (see the left picture of Figure 1). For more details about the experimental setup see the Supplementary Material. A third passive wheel keeps the body of the robot in a horizontal position. The active wheels are driven by motors that provide sensory feedback regarding the angle φ of the individual wheels. The working regime of the LEGO motors is finite, that is they respond to inputs $\tilde{M}\in \left[-{\tilde{M}}^{\text{max}},{\tilde{M}}^{\text{max}}\right]$. In order to comply with this constraint we mapped the simulated tangential force F^tan, defined by Equation (8), via

$\begin{array}{ll}\tilde{M}={\tilde{M}}^{\text{max}}tanh(F^{\text{tan}})& (9)\end{array}$

to the motor signal $\tilde{M}$. For an elastic response we used typical relative motor commands of the order of $M=\tilde{M}/{\tilde{M}}^{\text{max}}\in \left[-0.7,0.7\right]$. Absolute time was measured at the start of every control loop and compared with the last time the control loop was called. The such determined time difference Δt between two successive instances of the control loop was used for solving (7) via a straightforward Euler integration. On the average we had Δt ≈ 40ms.

The motor stalls for low motor powers, viz when $\left|\tilde{M}\right|\le 0.1{\tilde{M}}^{\text{max}}$, as a consequence of the internal friction of the gearing. A minimal torque is hence required for the robot to start moving. The overarching dynamical system, describing the body, the internal controller and the interactions between body and environment, allows for the generation of self-organized attractors (Sándor et al., 2015) that correspond to different motion patterns, the motor primitives (Ijspeert et al., 2002).

3. Results

3.1. Self-Organized Attractors As Motor Primitives

In the normal mode the robot disposes, as shown in Figure 2, of three possible states: stopped, forward and backward moving. For the forward and backward limit cycle locomotion the torque acting on the wheels is quasi-stationary, an observation that is consistent with the analytic treatment detailed out in section 4. Additionally, the robot may also rotate around its own axis, which happens when the two active wheels turn in opposite directions or when only one of the two wheels turns.

FIGURE 2

Figure 2. Motion primitives corresponding to non-moving and to forward and backward limit-cycle locomotion. Note that the motor stalls for |M| < M^thr ≈ 0.1 (gray area), viz when the torque M is unable to overcome the internal friction. Left: The measured output torque M for a = 4 and τ = 250ms as a function of the spring constant k. The measurements are performed after the robot settles in the forward (blue) or in the backward (green) attractor. For small k the robot stops moving and the tangential force vanishes. Right: An enlargement of the region k ∈ [0, 2] showing the measurements (blue/green dots) and the corresponding stable limit cycle (blue/green lines). A saddle-node bifurcation of limit-cycles (SNC) is likely to occur when the torque is counteracted by the internal friction. The resulting unstable limit cycles (dashed blue/green lines) are shown.

The measured speed of the robot in the forward and backward moving modes is v = 0.35m/s for k = 8, a = 4, and τ = 250ms, a setting that makes use of about 80–90% of the maximal power of the motor. In addition to the basic limit cycle attractors one finds for a = 4, k = 15, and τ = 1, 000ms a chaotic attractor, for which the motors switch irregularly between the destabilized fundamental modes of the individual wheels, as illustrated in Figure 3. For a video of the chaotic dynamics of the robot see the Supplementary Material. We did not attempt, for the case of the LEGO robot, to fully map the set of parameters for which a stable chaotic attractor exists, e.g., by evaluating the Lyapunov exponents in the context of navigation (Harter and Kozma, 2005). We note that chaotic attractors have been shown to be useful in the design and construction of spatial navigation models (Voicu et al., 2004).

FIGURE 3

Figure 3. Time series of the motor torques (9) in the chaotic mode, in relative units, for a = 4, τ = 1, 000ms and k = 15. Left: For the LEGO robot (click for movie). The light gray shaded area indicates the |M| < M^thr ≈ 0.1 region where the motor stalls. Right: For the analytic model (13), with F^tan given by the torque on right-hand side of $I\stackrel{·}{\omega }$. Here we took f = 0.5 for the friction and I = 0.05 for the moment of inertia.

3.2. Kick Control for Embodied Robots

The presence of coexisting attractors, viz of multistability (Pisarchik and Feudel, 2014), allows to switch between the basic modes without the need to modify the internal parameters of the system for the entire locomotion. The transitions between the individual attractors may be induced by external physical stimuli, such as collisions with other robots or with the environment (Martin et al., 2016).

A robot initialized in the forward moving mode is able to reverse direction when bouncing off a wall placed perpendicularly to the direction of locomotion, as illustrated in Figure 4. The reversal of direction is performed in this case autonomously, that is in absence of any additional control signals. It occurs because the forward mode gets destabilized for the duration of the collision, whereas the basin of attraction of the backward mode expands correspondingly. The flow in phase space is then drawn toward the backward attractor, where it stays after the forward attractor reemerges upon pulling away from the wall. Autonomous switching between forward and backward modes when colliding with obstacles occurs robustly, as we demonstrated by a series of experiments on rough surfaces (see Supplementary Video 3).

FIGURE 4

Figure 4. Collision induced switch of attractors. Middle: The time series of the relative torque M acting on the wheels for τ = 250ms, k = 2 and a = 4. The gray shaded region indicates the minimal torque needed to start the motor, M^thr = 0.1. The superimposed sketches illustrate a double-well potential with the minima corresponding to two coexisting attractors. The phase point (red ball) stays around the minima even in the presence of noise or small oscillations. The system is located in the backward attractor (B) when the robot collides with a wall and the forward attractor (F) is destabilized. The total torque M changes consequently its sign. Right, Left: The Lego robot before and after colliding with the wall (click for movie).

An alternative possibility to generate switches between coexisting attractors is to kick the phase point of the dynamical system to the basin of attraction of another attractor, termed here as kick control. This may be realized by applying short duration input stimuli. We present here three intuitive mechanisms, which may be classified, as discussed in section 2.1, as “frozen” and “quenched” kick control.

3.2.1. Frozen Kick Control

A reliable direction reversal may be induced by inverting membrane potentials via

$\begin{array}{ll}{x}_1\to -x_1,x_2\to -x_2,& (10)\end{array}$

at given time. Equation (10) induces an instantaneous change of the internal variables that does not affect the parameters of the one-neuron controller. It corresponds therefore, as defined in section 2.1, to frozen kick control. The respective time-series of the membrane potentials are shown in Figure 5. The reversal of the direction occurs fast, depending however on the state the actuator was in when the membrane potentials were inverted. It is furthermore noticeable, in particular following the second kick signal, that it may take half a second or more to fully settle into the reversed attractor. A relative phase slip may be induced in addition in between the two wheels, which are mechanically not precisely identical. The here considered variant (2) of frozen kick control is furthermore 100% reliable in direction reversal tests (for a demonstration see Supplementary Video 4). This is due to the fact that the sign flip of internal variables x_{1, 2} leads instantaneously to a motor torque of opposite direction, compare Equations (4, 8).

FIGURE 5

Figure 5. Kick controlling the LEGO Mindstorms robot. Shown are the time series of the membrane potentials (brown/orange and blue/cyan lines for x₁/x₂ of the left/right wheel), together with the normalized motor control $M=\tilde{M}/{\tilde{M}}^{max}$ (black/gray lines for left/right wheel). The parameters τ = 250ms, k = 8 and a = 4 lead to an angular frequency of ω/(2π) ≈ 2Hz. Top: Inverting all membrane potentials x_i, compare (10), with the times indicated by the vertical red lines (click for movie). Middle: Inverting all actual positions $x_i^{(a)}$, see (11), for three ticks of the updating cycle (roughly 90 ms, as indicated by the red vertical bars). Bottom: Adding a phase shift of Δφ = ±3π/4 to the measured angle of the wheels, see (12). The length of the control signal is here 4 ticks, corresponding to 120 ms. Note that the time to settle in the reverse limit cycle exceeds the duration of the kick signal. The relative phase of the left and the right wheel increases considerably after the second reversal.

3.2.2. Quenched Kick Control

We considered two variants of quenched kick control. For the first variant one substitutes the actual wheel positions $x_{1,2}^{(a)}$ by

$\begin{array}{ll}\begin{array}{ccc}{x}_1^{(a)}& \to & (1-2\beta )cos(\phi )\\ x_2^{(a)}& \to & (1-2\beta )sin(\phi )\end{array},\beta =\{\begin{array}{cc}0& \mathrm{\text{(control off)}}\\ 1& \mathrm{\text{(control on)}}\end{array}.& (11)\end{array}$

The new parameter β = β(t) is turned on for a finite control period Δt, as illustrated in Figure 5. This control procedure mimics (10) in the sense that the reversal of the membrane potentials is not achieved by a direct kick in the phase space of internal variables, but by a change of parameters. Compare Equation (7).

Real-world robots come with a control cycle that discretizes time. We consequently measure the time Δt during which β(t) is active in terms of control cycles (ticks). In Figure 6, we present the results of an experiment testing the reliability of (11), that is for the probability that the robot reverses direction for a given Δt. For the time-series shown in Figure 5 the robot received a kick signal for three ticks, viz for about 90 ms.

FIGURE 6

Figure 6. Reliability of quenched kick control procedures for the LEGO Mindstorms robot in terms of the relative success rate when testing direction reversal. The parameters are τ = 250ms, k = 8 and a = 4, with each data point corresponding to the average of 50 trials. The duration Δt of the control signal is given in terms of ticks, with a tick corresponding to the period of the control cycle of the Lego robot (about 30 ms). Maximal reliability is achieved for 3–5 ticks (90–120 ms), both for reversing the actual position x^(a) [compare (11), green circles] and when adding a phase-shift of Δφ = ±3π/4 to the angle of the wheel [as defined by (12), orange triangles]. A positive Δφ induces here a transition from forward to backward locomotion (reversely for negative Δφ).

For an alternative type of quenched kick control we consider with

$\begin{array}{ll}\begin{array}{ccc}{x}_1^{(a)}& \to & cos(\phi +\gamma \Delta \phi )\\ x_2^{(a)}& \to & sin(\phi +\gamma \Delta \phi )\end{array},\gamma =\{\begin{array}{cc}0& \mathrm{\text{(control off)}}\\ 1& \mathrm{\text{(control on)}}\end{array}& (12)\end{array}$

a shift in the sensory value of the angle of the robot. The corresponding reliability statistics are also shown in Figure 6. Both types of quenched kick control, as defined by Equations (11, 12), need to be applied for 3–5 control ticks, corresponding to 90–150 ms, for the robot to turn direction. For the experiment presented in Figure 5 we used 4 ticks when kicking the robot via (12).

3.3. Self-Organized Train of Cars

We used the LPZRobots simulations environment (Der and Martius, 2012) to simulate robots with two actuated wheels, as illustrated in Figure 7. The individual robots have a body mass of 0.1kg, wheel mass of 0.05kg, wheel radius of 3cm, body radius of 10cm and body height of 5cm. The dimensionless friction coefficient is 0.1. The cars are controlled as described in section 2.2, but this time only a single simulated transmission rod is employed. This is possible, as the motor of the simulated robots transits to an idle state in the absence of an input signal.

FIGURE 7

Figure 7. Snapshots from the LPZRobots simulations environment (Der and Martius, 2012) of two-wheeled car-like and train-like robots. Left: The two active wheels (black) are controlled as described in section 2.2. The passive support wheel (white) below the body (yellow) prevents the car from tipping over (click for movie). Right: A train robot created by connecting cars via passive torsion springs (click for movie).

The single two-wheel car follows intricate non-holonomic trajectories when put in an environment containing confining slopes. We also constructed trains composed of two-wheel cars coupled passively via torsion springs. All 10 wheels, for the case of the five-car train shown in Figure 7, are independent. One observes that the ten wheels coordinate their rotations speed and direction, reacting in a coordinated manner upon encountering objects in structured environment. The resulting locomotion of the train is a prime example of a self-organizing process (see Supplementary Videos 5, 6).

4. Analytic Modeling

The fundamental attractor of the individual wheels may be studied analytically when lumping the feedback of the environment into a single equation of motion that contains friction in terms of a friction force $\propto \text{}f\omega$. The resulting dynamical system is then

$\begin{array}{cc}\begin{array}{l}\tau {\dot{x}}_1=\cos \phi -x_1\hfill \\ \tau {\dot{x}}_2=\sin \phi -x_2\hfill \\ \dot{\phi }=\omega \hfill \\ I\dot{\omega }=k(2y(x_1)-1)\sin \phi -k(2y(x_2)-1)\cos \phi -f\omega ,\hfill \end{array}& (13)\end{array}$

where ω denotes the angular velocity and φ the angle of the wheel. The membrane potential of the horizontal and vertical controllers are, as illustrated in Figure 1, x₁ and x₂. The moment of inertia of the wheel is proportional to I, the spring constant by k, the friction coefficient by f, the time constant of the membrane potentials by τ and the transfer function y(x) of the controlling neurons by the sigmoidal y(x) = 1/(1 + exp(−ax)).

Note that x₁ and x₂ are internal variables of the robot, whereas φ (and consequently also ω) corresponds to the physical angle of the wheel and therefore to an environmental variable. This is because the body of the robot, including the wheels, are, from the perspective of the neural circuitry, part of the environment. We also point out that the motor power needs to exceed a certain threshold for the LEGO Mindstorms actuators to become active, as explained in section 2.3. This feature goes however beyond Equation (13).

4.1. Stationary Wheel

The system of four coupled differential equations (13), possesses eight trivial fixpoints, characterized by

$\begin{array}{ll}{\omega }_n^{*}=0,{\phi }_n^{*}=n\pi /4,n\in \left\{0,\dots ,7\right\},& (14)\end{array}$

of which the odd multiples of π/4 are always unstable. The even multiplies of π/4 are on the other side stable/unstable for small and large ratios of akτ/(2f), respectively, as we will show further below. The two-wheeled robot can hence be in 4 × 4 = 16 non-moving states corresponding to the 16 combinations of stable fixpoints of Equation (14) of the left and right wheel.

4.2. Homoclinic Route to Locomotion

In order to understand the transition from the fixpoint solutions (14) to locomotion we use

$\begin{array}{l}cos\phi (t^{\prime })\approx cos\phi (t)-sin\phi (t)\stackrel{·}{\phi }(t)(t^{\prime }-t)\\ =cos\phi (t)+sin\phi (t)\omega (t)(t-t^{\prime }),& (15)\end{array}$

which is valid for ωτ ≪ 1 and t − t′ ≪ τ, to expand the formal integral

$\begin{array}{ll}{x}_1(t)=\frac{1}{\tau }{\displaystyle {\int }_{-\infty }^t}d{t}^{\prime }cos\phi (t^{\prime })e^{-(t-t^{\prime })/\tau }\approx cos\phi (t)+\omega (t)\tau sin\phi (t)& (16)\end{array}$

of x₁(t). An equivalent expansion may be derived for x₂(t). One next expands the neural transfer functions occurring on the right-hand side of $\stackrel{·}{\omega }$,

$\begin{array}{ll}\begin{array}{ccc}y(x_1)& \approx & y(cos\phi )+ay(cos\phi )(1-y(cos\phi ))\omega \tau sin\phi \\ y(x_2)& \approx & y(sin\phi )-ay(sin\phi )(1-y(sin\phi ))\omega \tau cos\phi \end{array}& (17)\end{array}$

for small ωτ. The equations of motion (13) then take the form

$\begin{array}{ll}\stackrel{·}{\phi }=\omega ,I\stackrel{·}{\omega }=F(\phi )+\gamma (\phi )\omega ,& (18)\end{array}$

where F(φ) is a mechanical force and γ(φ) the coefficient of an adaptive friction. The respective expressions are

$\begin{array}{cc}F=2k\left[y(\cos \phi )\sin \phi -y(\sin \phi )\cos \phi \right]+k\left[\cos \phi -\sin \phi \right]& (19)\end{array}$

and

$\begin{array}{cc}\begin{array}{l}\gamma =2ka\tau [y(\cos \phi )(1-y(\cos \phi )){\sin }^2\phi \hfill \\ +y(\sin \phi )(1-y(\sin \phi )){\cos }^2\phi ]-f.\hfill \end{array}& (20)\end{array}$

Within this approximation, one finds that γ(φ) is negative for all φ ∈ [0, 2π] when

$\begin{array}{ll}k<k_c,k_c=\frac{2f}{a\tau }.& (21)\end{array}$

The system is purely dissipative when γ < 0, which implies that the fixpoints ${\phi }_{2n}^{*}=n\pi /2$ are stable for k < k_c. Locomotion is next achieved via a two-step process, as illustrated in Figure 8, when increasing the spring constant k beyond k_c.

FIGURE 8

Figure 8. Illustration of the homoclinic route to locomotion discussed in section 4.2. The phase-space plots (φ, ν = ω/(2π)) are for the analytic model (13), with a = 4, τ = 0.25, I = 0.05, and f = 0.5. The spring constant is k = 1.85/1.65/1.4 (top/middle/bottom). The fixpoints located at odd multiples of π/4 are saddles. Shown are stable (yellow/orange) and unstable (blue/green) manifolds, limit cycles (red), stable and unstable foci (filled/open circles) and some selected generic trajectories (black). Top: Limit-cycle locomotion, with the stable orbit winding around φ ∈ [−π, π]. Middle: Forth-and-back rolling with the angle φ of the wheel being limited to a finite range around multiples of π/2. Bottom: Stationary states with φ → nπ/2.

– The fixpoints ${\phi }_{2n}^{*}=n\pi /2$ undergo a supercritical Hopf bifurcation at akτ = 2f, viz when $\gamma ({\phi }_{2n}^{*})$ becomes positive. The angle φ of the wheel then oscillates around the previously stable fixpoint ${\phi }_{2n}^{*}$, with a trajectory that corresponds to a periodic forth-and-back motion of the robot.

– The amplitude of the limit cycle in φ will reach eventually, when k is further increased, the saddles at ${\phi }_{2n}^{*}=n\pi /2+\pi /4$. The limit cycle will then merge with the respective stable and unstable manifold of the saddle and undergo a Taken-Bogdanov-type (Gros, 2015) homoclinic bifurcation. Above this transition the four symmetry-related limit cycles around ${\phi }_{2n}^{*}=n\pi /2$ merge into a large cycle. The wheel then performs complete rotations.

We note that an equivalent merging of symmetry related limit cycles across a global bifurcation has been observed in a study of prototype dynamical systems (Sándor and Gros, 2015).

4.3. Constant Velocity Approximation

The angular moment ω becomes nearly constant for k far above the infinite-period transition. The solution of Eqution (7) obtained in the limit t → ∞ is given for the case of a constant angular velocity as

$\begin{array}{ll}{x}_1(t)=\frac{cos(\omega t)+\omega \tau sin(\omega t)}{1+{\omega }^2{\tau }^2},x_2(t)=\frac{sin(\omega t)-\omega \tau cos(\omega t)}{1+{\omega }^2{\tau }^2},& (22)\end{array}$

where we have used φ(t) = ωt. Assuming small amplitude oscillations for the membrane potential, ax_i ≪ 1, we can linearize the transfer function y(x) around y(0) = 1/2. The total tangential force F^tan defined by Equation (8) then becomes constant,

$\begin{array}{ll}{F}^{tan}=\frac{ak\omega \tau /2}{1+{\omega }^2{\tau }^2},\frac{ak\omega \tau /2}{1+{\omega }^2{\tau }^2}=f\omega ,{\omega }_{\pm }^{*}=\pm \sqrt{\frac{ak\tau -2f}{2f{\tau }^2}},& (23)\end{array}$

where the second equation corresponds to the balance between F^tan and the friction force fω in Equation (13). Locomotion vanishes in the constant-ω approximation for k < k_c = 2f/(aτ), viz at the critical spring constant k_c defined in (21).

The two symmetrical branches corresponding to the stable attractors of forward and backward motion can be seen in the experimentally constructed bifurcation diagram shown in the right panel of Figure 2. The internal friction forces of the motor induces in addition two symmetry-related saddle-node bifurcations of limit cycles, when reducing k, such that the torque M drops discontinuously to zero as k → k_c. The internal threshold of real-world motors impacts the route to chaos hence qualitatively. Compare section 4.2.

4.4. Numerical and Analytic Phase Diagram

In Figure 9, we present the phase diagram, as obtained by integrating (13) numerically. Four phases are found, a stationary fixpoint phase (S) at low values of kτ, a phase corresponding to forth-and-back (FB) motion, the limit-cycle locomotion phase (LLM) and, for large values of kτ, a region characterized by chaotic (C) locomotion. These phases may be characterized by the standard deviation $\text{std}(\omega )=\sqrt{\langle {\omega }^2\rangle -{\langle \omega \rangle }^2}$ of the angular frequency, which is elevated in the chaotic and in the FB phase, and small for LLM. The average angular frequency |〈ω〉| is in contrast highest for limit-cycle locomotion.

FIGURE 9

Figure 9. Left: The numerical phase diagram, as obtained by simulating (13) for a = 4, I = 0.05 and f = 0.5. Shown is the standard deviation $\text{std}(\omega )=\sqrt{\langle {\omega }^2\rangle -{\langle \omega \rangle }^2}$ of the angular frequency ω of the wheel (color coded), as averaged over time. Low values of std(ω) correspond to vanishing or near constant ω(t), as for the fixpoint solution (S) and for the limit-cycle locomotion (LLM). The transition to chaos (C) takes place through a crisis, with the route to locomotion occurring, as discussed in section 4.2, via an intermediate phase of forth-and-back rolling (FB). The inset enlarges the region k ∈ [0, 5] and τ ∈ [0, 0.5]. Right: The average angular frequency |〈ω〉| (color coded). The dashed lines correspond to the instability lines of the fixpoint solution, as obtained numerically (white) and analytically (gray), where the theoretical result is k_cτ_c = 2f/a. See Equation (21).

Also shown in Figure 9 is a comparison with the approximate estimate (21) of the transition between the stationary and the locomotive state, which is accurate when the transfer function y(x) can be linearized, viz in the limit of a vanishing FB phase. A typical time series of the motor torque within the chaotic phase is shown in the right panel of Figure 3.

5. Discussion

Robotic locomotion may be generated either via top-down control mechanisms or alternatively via self-organizing dynamical processes (Aguilar et al., 2016). In the first, more traditional approach, the “brain” of the agent is responsible for computing the control signals that drive the actuators, with error correction occurring through a high level evaluation of sensory measurements. The complexity of the control problem can however be reduced when robotic behavior is generated via self-organizing processes and local instabilities of the neural dynamics (Der and Martius, 2012).

Our work aims to reduce the theoretical and the computational constraints needed to design autonomous agents. Based on previous works on barrel- and sphere shaped robots (Sándor et al., 2015; Martin et al., 2016), we propose and study a novel actuator for wheeled robots. The actuator simulates the physics of the transmission rod used by classical steam engines, being controlled at the same time by only one or two rate-encoding neurons. Together with the wheel and with the proprioceptual environmental feedback, the controlling neurons form an overarching dynamical system that generates motion primitives in terms of stable attractors. The such produced self-organized fixpoints, limit cycles, and chaotic attractors, correspond to non-moving robots, to robots moving with constant speed and, respectively, to robots engaging in exploratory behaviors. All results are robust to noise present either in the environment or in the proprioceptual input stream (Martin et al., 2016).

A particular feature of the controller proposed here is that the direction of the movement, forward or backward, is selected by breaking time reversal symmetry. Studying two-wheeled robots we demonstrate that attractors corresponding to forward and backward motions coexist in the sensorimotor loop, allowing the robot to change direction autonomously when colliding with a wall. Switching between stable attractors can be achieved furthermore via a higher-order top-down control. Implementing frozen and quenched control signals, we are able to kick the robot reliably between distinct attractors, that is to kick the robot from one motor primitive into another motion primitive.

Examining in addition the bifurcation diagram leading from stationary states characterized by fixpoints in the phase space of the sensorimotor loop to limit-cycle locomotion, we find that two routes to locomotion exist: a single step process via a saddle node bifurcation of limit cycles, and a two-step scenario via a supercritical Hopf bifurcation followed by limit-cycle merging through distinct homoclinic bifurcations.

The here presented kick-control schemes demonstrate that simple impulse-like control signals are sufficient for creating complex behaviors whenever the motor primitives are given in terms of stable sensorimotor attractors. Hence one may speculate that kick control could also be used effectively in case of more complex robotic architectures, possibly in a combination with other types of control schemes (e.g., with the KA models; Harter and Kozma, 2005).

The concept of kick control also allows for a more general framework which is not necessarily neuro related. However, when it comes to the scalability of the proposed control scheme to robots with several dozens or even hundreds of degrees of freedoms and sensory channels, the most convenient underlying dynamical systems for the internal controller are adaptive neural networks with learning. The attractors there are then generalizations of the simple limit-cycle and chaotic attractors presented here, but the main idea of kicking the phase point from the basin of attraction of one attractor to another attraction domain remains the same.

Finally, we believe that the proposed model for generating attractors for locomoting robots and controlling their motion by kicking the phase point to their respective basins of attraction may also be used for teaching dynamical systems in advanced high school physics courses. The Lego robots allow for interactive demonstrations, e.g., in lab activities, of how attractors may be used in real-world applications, hence providing an intuitive understanding of the terminology and underlying phenomena.

Author Contributions

The manuscript was written by BS and CG. The experiments were conceived and designed by CG, BS, TK, MN, and LM. Lego experiments performed mainly by TK and MN. Simulations performed by LM. The data was analyzed by BS and CG with calculations proposed by MN, and plots produced by BS and TK.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The support of the German Science Foundations (DFG) and discussions with Komal Bhattacharyya are acknowledged. The work of BS was funded by the Content Pedagogy Research Program of the Hungarian Academy of Sciences.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnbot.2018.00040/full#supplementary-material

References

Aguilar, J., Zhang, T., Qian, F., Kingsbury, M., McInroe, B., Mazouchova, N., et al. (2016). A review on locomotion robophysics: the study of movement at the intersection of robotics, soft matter and dynamical systems. Rep. Prog. Phys. 79:110001. doi: 10.1088/0034-4885/79/11/110001

PubMed Abstract | CrossRef Full Text | Google Scholar

Conradt, J., Galluppi, F., and Stewart, T. C. (2015). Trainable sensorimotor mapping in a neuromorphic robot. Robot. Auton. Syst. 71, 60–68. doi: 10.1016/j.robot.2014.11.004

CrossRef Full Text | Google Scholar

Das, T., Kar, I., and Chaudhury, S. (2006). Simple neuron-based adaptive controller for a nonholonomic mobile robot including actuator dynamics. Neurocomputing 69, 2140–2151. doi: 10.1016/j.neucom.2005.09.013

CrossRef Full Text | Google Scholar

Der, R., and Martius, G. (2012). The Playful Machine: Theoretical Foundation and Practical Realization of Self-Organizing Robots, Vol. 15. Berlin; Heidelberg: Springer Science & Business Media.

Google Scholar

Ghazi-Zahedi, K., Langer, C., and Ay, N. (2017). Morphological computation: synergy of body and brain. Entropy 19:456. doi: 10.3390/e19090456

CrossRef Full Text | Google Scholar

Gros, C. (2014). “Generating functionals for guided self-organization,” in Guided Self-Organization: Inception, ed M. Prokopenko (Springer), 53–66.

Google Scholar

Gros, C. (2015). Complex and Adaptive Dynamical Systems: A Primer. Berlin; Heidelberg: Springer.

Harter, D., and Kozma, R. (2005). Chaotic neurodynamics for autonomous agents. IEEE Trans. Neural Netw. 16, 565–579. doi: 10.1109/TNN.2005.845086

PubMed Abstract | CrossRef Full Text | Google Scholar

Hennig, M. H. (2013). Theoretical models of synaptic short term plasticity. Front. Comput. Neurosci. 7:45. doi: 10.3389/fncom.2013.00045

CrossRef Full Text | Google Scholar

Ijspeert, A. J., Nakanishi, J., and Schaal, S. (2002). “Learning attractor landscapes for learning motor primitives,” in Advances in Neural Information Processing Systems (Cambridge, MA: MIT Press), 1547–1554.

Google Scholar

Jimenez-Fernandez, A., Jimenez-Moreno, G., Linares-Barranco, A., Dominguez-Morales, M. J., Paz-Vicente, R., and Civit-Balcells, A. (2012). A neuro-inspired spike-based PID motor controller for multi-motor robots with low cost FPGAs. Sensors 12, 3831–3856. doi: 10.3390/s120403831

PubMed Abstract | CrossRef Full Text | Google Scholar

Kozłowski, K. and Pazderski, D. (2004). Modeling and control of a 4-wheel skid-steering mobile robot. Int. J. Appl. Math. Comp. Sci. 14, 477–496.

Google Scholar

Kubisch, M., Werner, B., and Hild, M. (2011). “Using co-existing attractors of a sensorimotor loop for the motion control of a humanoid robot,” in IJCCI (NCTA) (Paris), 385–390.

Google Scholar

Linkerhand, M., and Gros, C. (2013). Generating functionals for autonomous latching dynamics in attractor relict networks. Sci. Rep. 3:2042. doi: 10.1038/srep02042

PubMed Abstract | CrossRef Full Text | Google Scholar

Mandow, A., Martinez, J. L., Morales, J., Blanco, J. L., Garcia-Cerezo, A., and Gonzalez, J. (2007). “Experimental kinematics for wheeled skid-steer mobile robots,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE) (San Diego, CA), 1222–1227.

Google Scholar

Martin, L., Sándor, B., and Gros, C. (2016). Closed-loop robots driven by short-term synaptic plasticity: emergent explorative vs. limit-cycle locomotion. Front. Neurorobot. 10:12. doi: 10.3389/fnbot.2016.00012

PubMed Abstract | CrossRef Full Text | Google Scholar

Martius, G., Der, R., and Ay, N. (2013). Information driven self-organization of complex robotic behaviors. PLoS ONE 8:e63400. doi: 10.1371/journal.pone.0063400

PubMed Abstract | CrossRef Full Text | Google Scholar

Pfeifer, R., Lungarella, M., and Iida, F. (2007). Self-organization, embodiment, and biologically inspired robotics. Science 318, 1088–1093. doi: 10.1126/science.1145803

PubMed Abstract | CrossRef Full Text | Google Scholar

Pisarchik, A. N., and Feudel, U. (2014). Control of multistability. Phys. Rep. 540, 167–218. doi: 10.1016/j.physrep.2014.02.007

CrossRef Full Text | Google Scholar

Prokopenko, M., Boschetti, F., and Ryan, A. J. (2009). An information-theoretic primer on complexity, self-organization, and emergence. Complexity 15, 11–28. doi: 10.1002/cplx.20249

CrossRef Full Text | Google Scholar

Sándor, B., and Gros, C. (2015). A versatile class of prototype dynamical systems for complex bifurcation cascades of limit cycles. Sci. Rep. 5:12316. doi: 10.1038/srep12316

PubMed Abstract | CrossRef Full Text | Google Scholar

Sándor, B., Jahn, T., Martin, L., and Gros, C. (2015). The sensorimotor loop as a dynamical system: how regular motion primitives may emerge from self-organized limit cycles. Front. Robot. AI 2:31. doi: 10.3389/frobt.2015.00031

CrossRef Full Text | Google Scholar

Tani, J., and Ito, M. (2003). Self-organization of behavioral primitives as multiple attractor dynamics: a robot experiment. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 33, 481–488. doi: 10.1109/TSMCA.2003.809171

CrossRef Full Text | Google Scholar

Todorov, E. (2004). Optimality principles in sensorimotor control. Nat. Neurosci. 7, 907–915. doi: 10.1038/nn1309

PubMed Abstract | CrossRef Full Text | Google Scholar

Toutounji, H., and Pasemann, F. (2014). Behavior control in the sensorimotor loop with short-term synaptic dynamics induced by self-regulating neurons. Front. Neurorobot. 8:19. doi: 10.3389/fnbot.2014.00019

PubMed Abstract | CrossRef Full Text | Google Scholar

Voicu, H., Kozma, R., Wong, D., and Freeman, W. J. (2004). Spatial navigation model based on chaotic attractor networks. Connect. Sci. 16, 1–19. doi: 10.1080/09540090410001664641

CrossRef Full Text | Google Scholar