Event Abstract

Back to Event

Multiple Time Scales Recurrent Neural Network for Complex Action Acquisition

Martin Peniak1*, Davide Marocco1, Jun Tani2, Yuichi Yamashita2, Kerstin Fischer3 and Angelo Cangelosi1
  • 1 The University of Plymouth, United Kingdom
  • 2 RIKEN, Brain Science Institute, Japan
  • 3 University of Southern Denmark, Denmark

Introduction

Humans are able to acquire many skilled behaviors during their life-times. The learning of complex behaviours is achieved through a constant repetition of the same movements over and over, with certain components segmented into reusable elements known as motor primitives. These motor primitives are then flexibly reused and dynamically integrated into novel sequences of actions [1], [2]. For example, the action of lifting an object can be broken down into a combination of multiple motor primitives. Some motor primitives would be responsible for reaching the object, some for grasping it and some for lifting it. These primitives are represented in a general manner and should therefore be applicable to objects with different properties.
Yamashita and Tani [3] were inspired by the latest biological observations of the brain to develop a completely new model of action sequence learning known as Multiple Timescales Recurrent Neural Network (MTRNN). The MTRNN attempts to overcome the generalisation-segmentation problem of previous action learning models based on explicitly structured functional hierarchies such as MOSAIC [4] or the mixture of multiple recurrent neural network expert systems [5]. This is achieved through the realisation of functional hierarchy that is neither based on separate modules nor on a structural hierarchy but rather on multiple timescales of neural activities implementation of which was inspired by the biological findings (e.g. [6], [7]).

This paper presents novel results of complex action learning based on an extended MTRNN model. The results showed that the system was able to learn eight different sensorimotor patters, which form the basis of our next experiments on action and language compositionality.

Method

The preliminary experiment presented in the paper implements the extended MTRNN model embodied in the iCub humanoid robot (www.icub.org) [8]. The model was implemented as part of Aquila cognitive robotics toolkit [9] that makes use of massively parallel GPU devices that significantly outperform standard CPU processors on parallel tasks. This allowed for the extension of the previously used MTRNN model [3] with a higher number of neurons and sensorimotor sequences.

The MTRNN's core is based on a continuous time recurrent neural network characterised by the ability to preserve its internal state and hence exhibit complex dynamics. The system receives sparsely encoded proprioceptive input from the robot, which is used to predict next sensorimotor states and it therefore acts as a forward kinematics model.

The neural activities were calculated following the classical firing rate model where each neuron's activity is given by the average firing rate of the connected neurons. In addition to this, the MTRNN model implements a leaky integrator and therefore the state of every neuron is not only defined by the current synaptic inputs but also considers its previous activations. The extent to which the previous activities of neurons affect their current states is defined by the decay rate parameter τ. Therefore, when the neurons are set with large τ values their activities will be changing more slowly over time as compared to those neurons set with smaller τ values.

The MTRNN needs to be trained via an algorithm that considers its complex dynamics changing through time and for this reason we used a backpropagation through-time (BPTT) algorithm as it has been previously demonstrated to be effective with this recursive neural architecture [3].

A self-organising map (SOM) was used as the input to the MTRNN system to help preserve the topological relations in the multidimensional input space by reducing the possible overlaps between various sensorimotor sequences. The SOM was trained offline using a conventional unsupervised learning algorithm implemented in Aquila SOM module.


Experiments and Results

This section presents results of the initial testing of the MTRNN model on the iCub humanoid robot. The experimental task required the MTRNN system to learn the eight behavioural patterns (slide box left/right, swing box, lift box up/left/right, push/pull box). The Sequence Recorder module of Aquila was used to record these sensorimotor patterns while the experimenter was guiding the robot by holding its arms and performing the actions.

In this experiment, 256 input-output neurons were set to τ = 2 and hidden neurons consisted of two different categories where each had a different time integration constant. The first category comprise of 60 fast neurons with τ = 5 and the second of 20 slow neurons set to τ = 70. These two categories are attempting to capture the dynamics of complex behavioural patterns by flexible recombination of motor primitives into novel sequences of actions. The network is fully connected and hence every neuron is connected to every other neuron including itself. There is one exception where the slow neurons are not directly connected to the input-output layer but rather indirectly via the fast neurons.

Both SOM and MTRNN were trained on a data set consisting of eight different behavioural patterns. Every behaviour was recorded three times with slight variations that involved 5cm offsets with respect to the center of the object. This was done to achieve smooth representations of the input space and to reduce the errors incurred during the SOM transformations. Thousands of sensorimotor states were recorded and used to train the SOM prior to the MTRNN training, which only used the original sequence (without offsets) for each behaviour.

Five different learning trials were conducted, where each trial was initialised with a different seed that was used to generate random numbers for synaptic connections. The BPTT algorithm was set to run for one million iterations with the learning rate set to 0.015 and sigma parameter set to 0.0045. This computationally intensive training was possible through the utilisation of a cluster of NVIDIA Tesla and Fermi GPU cards as well as the Aquila CUDA compliant module.

At the end of the training, the learned neural network was tested on the iCub in the same setup as that during the tutoring part. The results showed that the MTRNN system was able to replicate all the eight sequences while successfully manipulating the object.


Conclusion and Future Work

We have showed that the MTRNN model was able to learn eight different behavioural sequences. These constitute the motor primitives for ongoing experiments for the learning of action and language compositionality that will be addressing a specific linguistic hypothesis first proposed by the cognitive psychologist and linguist Michael Tomasello. The hypothesis, which is also known as the verb island theory, predicts that verbal argument structures are learned on a purely item-specific basis [10].

The first planned experiment will investigate the role of semantic similarities between different words during early language acquisition. In particular, the hypothesis addressed by this experiment is whether a generalisation to unheard sentences is easier in condition where all learned events are of the same semantic type. Though conceptually simple, this experiment will constitute the first viable extension of the already conducted research within the iTalk Project. In addition, these problems are also discussed in child development research and therefore this work could provide useful insights.

The extension of the experiment will investigate the effects of using different learning techniques such as holistic, scaffolded and parallel learning. There are several other possibilities for farther experiments on which we are yet to agree, however, the experiments outlined in this section present an important step towards expanding our current knowledge of action-language integration as well as the acquisition of more complex grammatical constructions.

Acknowledgements

This work was supported by the EU Integrating Project - ITALK (214886) within the FP7 ICT programme - Cognitive Systems, Interaction and Robotics.

References

[1] K. A. Thoroughman and R. Shadmehr, “Learning of action through adaptive combination of motor primitives,” Science, vol. 407, pp. 742–747, 2000.

[2] A. d’Avella, A. Portone, L. Fernandez, and F. Lacquaniti, “Control of fast-reaching movements by muscle synergy combinations,” Neuroscience, vol. 26, no. 30, pp. 7791–7810, 2006.

[3] Y. Yamashita and J. Tani, “Emergence of functional hierarchy in a multiple timescale neural network model: a humanoid robot experiment,” PLoS Computational Biology, vol. 4, no. 11, 2008.

[4] D. M. Wolpert and M. Kawato, “Multiple paired forward and inverse models for motor control,” Neural Networks, pp. 1317–1329, 1998.

[5] J. Tani and S. Nolfi, “Learning to perceive the world as articulated: an approach for hierarchical learning in sensory–motor systems,” Neural Networks, p. 1131–1141, 1999.

[6] R. Huys, A. Daffertshofer, and P. J. Beek, “Multiple time scales and multiform dynamics in learning to juggle,” Motor Control, vol. 8, pp. 188–212, 2004.

[7] F.Varela,J.P.Lachaux,E.Rodriguez,andJ.Martinerie,“Thebrainweb:phasesynchronizationandlarge-scaleintegration,”NatureReviewsNeuroscience, vol. 2, pp. 229–239, 2001.

[8] G.Metta,L.Natale,F.Nori,G.Sandini,D.Vernon,L.Fadiga,C.vonHofsten,K.Rosander,M.Lopes,J.Santos-Victor,A.Bernardino,andL.Montesano, The icub humanoid robot: An open-systems platform for research in cognitive development,” Neural Networks, vol. 23, no. 8-9, pp. 1125–1134, 2010.

[9] M. Peniak, A. Morse, C. Larcombe, S. Ramirez-Contla, and A. Cangelosi, “Aquila: An open-source gpu-accelerated toolkit for cognitive robotics research,” in International Joint Conference on Neural Networks (IJCNN) 2011, San Jose, California, in press.

[10] M. Tomasello, Constructing a language. Cambridge, Massachusetts: Harvard University Press, 2003.

Keywords: action acquisition, Aquila, iCub, iTalk, MTRNN, Neural Network, timescales

Conference: IEEE ICDL-EPIROB 2011, Frankfurt, Germany, 24 Aug - 27 Aug, 2011.

Presentation Type: Poster Presentation

Topic: Architectures

Citation: Peniak M, Marocco D, Tani J, Yamashita Y, Fischer K and Cangelosi A (2011). Multiple Time Scales Recurrent Neural Network for Complex Action Acquisition. Front. Comput. Neurosci. Conference Abstract: IEEE ICDL-EPIROB 2011. doi: 10.3389/conf.fncom.2011.52.00009

Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters.

The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated.

Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed.

For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions.

Received: 24 Jun 2011; Published Online: 12 Jul 2011.

* Correspondence: Mr. Martin Peniak, The University of Plymouth, Plymouth, United Kingdom, martin.peniak@plymouth.ac.uk