AUTHOR=Rostami Amirhossein , Vogginger Bernhard , Yan Yexin , Mayr Christian G. 

TITLE=E-prop on SpiNNaker 2: Exploring online learning in spiking RNNs on neuromorphic hardware

JOURNAL=Frontiers in Neuroscience

VOLUME=Volume 16 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.1018006

DOI=10.3389/fnins.2022.1018006

ISSN=1662-453X

ABSTRACT=In recent years, the application of deep learning models at the edge has gained attention. Typically, artificial neural networks (ANNs) are trained on GPUs and optimized for efficient execution on edge devices. Training ANNs directly at the edge is the next step with many applications such as the adaption of models to specific situations like changes in environmental settings or optimization for individuals, e.g. optimization for speakers for speech processing. Also, local training can preserve privacy.
Over the last few years, many algorithms have been developed to reduce memory footprint and computation.
A specific challenge to train recurrent neural networks (RNNs) for processing sequential data is the need for the Back Propagation Through Time (BPTT) algorithm to store the network state of all time steps. This limitation is resolved by the biologically-inspired e-prop approach for training Spiking Recurrent Neural Networks (SRNNs). We implement the e-prop algorithm on a prototype of the SpiNNaker 2 neuromorphic system. A parallelization strategy is developed to split and train networks on the ARM cores of SpiNNaker 2 to make efficient use of both memory and compute resources.
We trained an SRNN from scratch on SpiNNaker 2 in real-time on the Google Speech Command dataset for keyword spotting.
We achieve an accuracy of 91.12\% while requiring only 680KB of memory for training the network with 25K weights.
Compared to other spiking neural networks with equal or better accuracy, our work is significantly more memory-efficient.
In addition, we perform a memory and time profiling of the e-prop algorithm. This is used on the one hand to discuss whether e-prop or BPTT is better suited for training a model at the edge and on the other hand to explore architecture modifications to SpiNNaker 2 to speed up online learning.