AUTHOR=Jordan Ian D. , Sokół Piotr Aleksander , Park Il Memming 

TITLE=Gated Recurrent Units Viewed Through the Lens of Continuous Time Dynamical Systems

JOURNAL=Frontiers in Computational Neuroscience

VOLUME=Volume 15 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2021.678158

DOI=10.3389/fncom.2021.678158

ISSN=1662-5188

ABSTRACT=In recent years, the efficacy of using artificial recurrent neural networks to model cortical dynamics has been a topic of interest. Gated recurrent units (GRUs) are specialized memory elements for building these recurrent neural networks. Despite their incredible success in natural language, speech, video processing, and extracting dynamics underlying neural data, little is understood about the specific dynamics representable in a GRU network, and how these dynamics play a part in performance and generalization.
As a result, it is both difficult to know a priori how successful a GRU network will perform on a given task, and also their capacity to mimic the underlying behavior of their biological counterparts. Using a continuous time analysis, we gain intuition on the inner workings of GRU networks. We restrict our presentation to low dimensions, allowing for a comprehensive visualization. We found a surprisingly rich repertoire of dynamical features that includes stable limit cycles (nonlinear oscillations), multi-stable dynamics with various topologies, and homoclinic bifurcations. At the same time GRU networks are limited in their inability to produce continuous attractors, which are hypothesized to exist in biological neural networks. We contextualize the usefulness of different kinds of observed dynamics and support our claims experimentally.