# METASTABLE DYNAMICS OF NEURAL ENSEMBLES

EDITED BY: Emili Balaguer-Ballester, Ruben Moreno-Bote, Gustavo Deco and Daniel Durstewitz PUBLISHED IN: Frontiers in Systems Neuroscience

#### *Frontiers Copyright Statement*

*© Copyright 2007-2018 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.*

*The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.*

*Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.*

*Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.*

*As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.*

> *All copyright, and all rights therein, are protected by national and international copyright laws.*

*The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.*

ISSN 1664-8714 ISBN 978-2-88945-437-2 DOI 10.3389/978-2-88945-437-2

# About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

## Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

## Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

# What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# **METASTABLE DYNAMICS OF NEURAL ENSEMBLES**

Topic Editors:

**Emili Balaguer-Ballester,** Bournemouth University, United Kingdom; Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, Germany

**Ruben Moreno-Bote,** Universitat Pompeu Fabra, Parc Sanitari Sant Joan de Déu, Spain **Gustavo Deco,** Universitat Pompeu Fabra, Spain; Institució Catalana de la Recerca i Estudis Avançats (ICREA), Spain; Max Planck Institute for Human Cognitive and Brain Sciences, Germany; Monash University, Australia

**Daniel Durstewitz,** Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, Central Institute of Mental Health, and Heidelberg University, Germany

Image: whitehoune/Shutterstock.com

A classical view of neural computation is that it can be characterized in terms of convergence to attractor states or sequential transitions among states in a noisy background. After over three decades, is this still a valid model of how brain dynamics implements cognition?

This book provides a comprehensive collection of recent theoretical and experimental contributions addressing the question of stable versus transient neural population dynamics from complementary angles. These studies showcase recent efforts for designing a framework that encompasses the multiple facets of metastability in neural responses, one of the most fascinating topics currently in systems and computational neuroscience.

**Citation:** Balaguer-Ballester, E., Moreno-Bote, R., Deco, G., Durstewitz, D., eds. (2018). Metastable Dynamics of Neural Ensembles. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-437-2

# Table of Contents


*135 Decomposition of spontaneous brain activity into distinct fMRI co-activation patterns*

Xiao Liu, Catie Chang and Jeff H. Duyn

*146 Cortical Variability and Challenges for Modeling Approaches* Emili Balaguer-Ballester

# Editorial: Metastable Dynamics of Neural Ensembles

Emili Balaguer-Ballester 1,2 \*, Ruben Moreno-Bote<sup>3</sup> , Gustavo Deco4,5,6,7 and Daniel Durstewitz <sup>8</sup>

<sup>1</sup> Department of Computing and Informatics, Faculty of Science and Technology, Bournemouth University, Poole, United Kingdom, <sup>2</sup> Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, Mannheim, Germany, <sup>3</sup> Center for Brain and Cognition and Department of Information and Communications Technologies, Pompeu Fabra University, Research Unit, Parc Sanitari Sant Joan de Déu, Barcelona, Spain, <sup>4</sup> Center for Brain and Cognition, Computational Neuroscience Group, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain, <sup>5</sup> Institució Catalana de la Recerca i Estudis Avançats (ICREA), Barcelona, Spain, <sup>6</sup> Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany, <sup>7</sup> School of Psychological Sciences, Monash University, Melbourne, VIC, Australia, <sup>8</sup> Department of Theoretical Neuroscience, Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany

Keywords: metastability, trial-to-trial variability, transient dynamics, attractor dynamics, neural noise

#### **Editorial on the Research Topic**

#### **Metastable Dynamics of Neural Ensembles**

A classical view of neural computation is that it can be characterized in terms of convergence to fixed-point-type attractor states (representing for instance memory patterns in Hopfield, 1982) or limit-cycle-like sequential transitions among states (mapping e.g., motor or syntactical sequences in Elman, 1990). After over three decades, is this still a valid model of how brain dynamics implements cognition? The idea that neuro-computational dynamics is mainly deterministically driven by convergence to emergent stable states in a synaptic/network noisy background has been lively debated, and recently challenged both empirically and by computational work. This question touches on the very basics of our understanding of neural computation; and hence it is one of the most exciting topics currently in systems and computational neuroscience.

This e-book comprises a comprehensive collection of recent theoretical and experimental contributions addressing the question of stable versus transient neural population dynamics, and its implications for the observed variability in neural activity, from diverse, complementary angles.

# METASTABILITY IN MODELS

A connecting theme for the multiple contemporary views on metastability in the brain was proposed first by Tognoli and Kelso. In their foundational approach, the authors discuss classical and recent views on how information transfer between brain regions could be accomplished through synchronization and collective neural responses. They frame these ideas in terms of the coordination dynamics concept, potentially a key aspect for understanding metastability in neuronal populations.

Metastability and its possible functional role both within and outside of behavioral task contexts is further addressed in four specific modeling approaches situated at different spatial scales, ranging from macro/mesoscopic levels (Schwappach et al.; Stratton and Wiles; Aguilera et al.) to a biophysically detailed level of neuronal systems description (Mazzucato et al.).

The balance between global segregation and integration at a macroscopic scale is theoretically analyzed by Stratton and Wiles. They propose a computational model focused on how the thalamocortical loop may underlie long-range segregation between brain regions, producing metastable

#### Edited by:

Mikhail Lebedev, Duke University, United States

#### Reviewed by:

Hiroshi Okamoto, RIKEN Brain Science Institute (BSI), Japan

#### \*Correspondence:

Emili Balaguer-Ballester eb-ballester@bournemouth.ac.uk

Received: 27 November 2017 Accepted: 22 December 2017 Published: 26 January 2018

#### Citation:

Balaguer-Ballester E, Moreno-Bote R, Deco G and Durstewitz D (2018) Editorial: Metastable Dynamics of Neural Ensembles. Front. Syst. Neurosci. 11:99. doi: 10.3389/fnsys.2017.00099 responses observed at large spatial scales. Metastability at macroscopic levels could also stem from sensimotor interactions, as suggested by Aguilera et al. These authors designed a new theoretical framework and implemented it in an agent-based model which interacts with the environment. According to this model, metastability arises from the dynamics of sensori-motor feedback interactions, beyond what would be expected from considering brain activity just in isolation.

At mesoscopic scales, neural population models have been constructed that produce metastability through attracting chains of heteroclinic orbits, generating transient dynamics through a sequence of saddle points along which one or several axes are stable (the stable subspace). Following up on this theory, Schwappach et al. demonstrate, using a novel neural field model, how such heteroclinic subspaces can account for part of the observed trial-to-trial variability at the mesoscopic level. Hence, this variability may partly stem from sources other than neuronal or synaptic noise.

At microscopic (biophysical) scales, using a clustered spiking model (in which connectivity patterns are heterogeneous) which exhibits metastable states, Mazzucato et al. show that variation in neuronal ensemble activity may be confined to small subspaces of the whole state space spanned by all the individual units' firingrates. Moreover, the dimensionality of these subspaces is smaller during stimulus-evoked activity than in the absence of a task. This is in line with empirical studies which report the reduction of neuronal variability upon stimulus presentation (Churchland et al., 2010).

## EMPIRICAL STUDIES

Metastability was also addressed in four studies which provide novel analytical tools and empirical evidence. Tošic et al. ´ proposed a new data analysis technique to identify metastability empirically, which was used to infer metastable states in local field potentials evoked by visual stimuli in anesthetised ferrets.

Interestingly, visual scan paths (Wilkinson and Metta) reveal complex dynamics which possibly reflects underlying metastable neural activity. In Wilkinson and Metta, the authors proposed a theoretical framework, termed the singularity hypothesis, which relies on transient spiral waves which govern persistent neural activity states underlying oculomotor postural control. In general, motor control strategies may be represented in neuronal

# REFERENCES


activity patterns as a complex, distributed spatiotemporal code, which may not be revealed by looking just at neuronal firing rates within recorded ensembles. This is shown in Mao et al. who study behaving rats performing a directional choice task, using a nonlinear decoder to demonstrate how spatiotemporal activity patterns in motor areas increasingly discriminate the animal's choices as learning progresses.

Finally, spatiotemporal patterns generated by synchronized spontaneous activity in the idle brain are analyzed at different spatial scales in two studies (Liu et al.; Yada et al.) which further illustrate the rich diversity in methodological approaches to the empirical identification of metastable and transient dynamics. Specifically, Yada et al. propose that repeating spatiotemporal patterns emerging during synchronized bursts of activity in vitro may have their origin in specific sub-populations that become sequentially active in a reproducible temporal order. This result suggests an orchestrated activation of such ensembles that depends on the global state of the network, consistent with spontaneous transitions between metastable states.

At a macroscopic level, Liu et al. develop a new method for identifying co-activity patterns of fMRI responses which is robust to non-stationarity. In this study, a novel type of cluster analysis suggests a richer repertoire of co-activation states beyond the resting-state networks identified previously, and hence perhaps a more specialized functional organization.

In summary, this book provides a comprehensive collection of current modeling and data analysis approaches related to the metastable behavior of cortical ensembles. These studies showcase recent efforts for designing a fundamental framework that encompasses the multiple facets of metastability in neural responses, beyond the original use of the concept in the context of statistical mechanics. To conclude, the last chapter of the book reflects on this plethora of approaches and connects them with the question of the functional role of experimentally observed trial-to-trial variability (Balaguer-Ballester), one of the most intriguing topics currently in systems neuroscience (Moreno-Bote, 2014).

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Balaguer-Ballester, Moreno-Bote, Deco and Durstewitz. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Enlarging the scope: grasping brain complexity

# **Emmanuelle Tognoli <sup>1</sup>\* and J. A. Scott Kelso1,2**

<sup>1</sup> The Human Brain and Behavior Laboratory, Center for Complex Systems and Brain Sciences, Florida Atlantic University, Boca Raton, FL, USA 2 Intelligent System Research Centre, University of Ulster, Derry, Northern Ireland

#### **Edited by:**

Emili Balaguer-Ballester, Bournemouth University and Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, UK

#### **Reviewed by:**

Nicholas M. Dotson, Montana State University, USA Karl Friston, University College London, UK

#### **\*Correspondence:**

Emmanuelle Tognoli, The Human Brain and Behavior Laboratory, Center for Complex Systems and Brain Sciences, Florida Atlantic University, 777 Glades Road, Boca Raton, FL-33431, USA e-mail: tognoli@ccs.fau.edu

To further advance our understanding of the brain, new concepts and theories are needed. In particular, the ability of the brain to create information flows must be reconciled with its propensity for synchronization and mass action. The theoretical and empirical framework of Coordination Dynamics, a key aspect of which is metastability, are presented as a starting point to study the interplay of integrative and segregative tendencies that are expressed in space and time during the normal course of brain and behavioral function. Some recent shifts in perspective are emphasized, that may ultimately lead to a better understanding of brain complexity.

**Keywords: metastability, brain coordination dynamics, integration, segregation, functional connectivity**

# **THEORIES OF THE BRAIN: A CONCISE HISTORY**

How does the brain work? This nagging question is an habitué from the top ten lists of enduring problems in Science's grand challenges. Grasp this paradox: how is one human brain—a chef d'oeuvre of complexity honed by Nature—ever to reach such a feast as to understand itself? Where one brain may fail at this notorious philosophical riddle, may be a strong and diversely-skilled army of brains may come closer. Understanding of the local principles at play has emerged due to the combined efforts of many scientists: neurons talk to their partners by teasing them with charged particles of either excitatory or inhibitory effect, as Nobel laureate Sir John Eccles demonstrated (Eccles, 1966). Targeted release of ions was later shown at sites that seem designed for the exchange of information: typically the axonal termination of the emitting neuron facing the dendrites of a receiving partner (Hodgkin and Huxley, 1952). Many of those two-some neural interlocutors build into a reticulum with remarkable emergent properties. A booming network science followed, generalizing microscale principles on a large-scale. David Rumelhart and James McClelland, Stephen Grossberg and many others pursued this connectionist endeavor (Grossberg, 1976; Rumelhart and McClelland, 1986). Putting function first, they aimed to model specific aspects of human cognition and behavior such as visual perception or language. Yet others, such as Olaf Sporns have devoted much effort to the neurobiological fidelity of their inquiries, conceiving behavior as an emergent phenomenon from an appropriate connectional design (Sporns et al., 2004). The latter may be probed both with theoretical connectivity models where brain complexity is carefully thought through (Sporns et al., 2000) along with empirically-derived models that borrow their connectional blueprints from images of "real" brains (Honey et al., 2010). Though brief, this historical overture suggests that the brain has demonstrable mechanisms for both point-to-point communication of information—most obviously at the microscale—as well as emergent phenomena arising from network properties at the meso- and macroscale.

# **NEURONAL RELAYS AND THE PROPAGATION OF INFORMATION**

The principle of synaptic transmission proved to be pictureperfect for a theory of communication, boosted by the influential work published in 1948 by Claude Shannon (Shannon, 1948). Transfer of information became a principal tenet of brain function, and theories went so far as to conceive of "centers" (Charcot, 1878; see perspectives in Catani and ffytche, 2005) as final destinations for information to be communicated (the concept has now retreated, although it remains perniciously present in neuroscientists' conceptions of brain hierarchies<sup>1</sup> (e.g., Meunier et al., 2010); an alternate view is that it is the journey,

<sup>1</sup>Clearly, the brain is not a diffuse web of randomly coupled dynamical systems. It is a highly structured organ that has been crafted by evolution. The outcome of phylogenesis, a slow timescale pattern-forming process (Bressler and Tognoli, 2006) is the progressive emergence of functional specialization, where some brain regions take on a more integrated role than others, a phenomenon that has been named hierarchical organization (see Robert, 1999; Zilles et al., 2002 for anatomo-functional patterning, see also Honey et al., 2007 for a related network perspective). The notion of hierarchy is popular in neuroscience and may be relevant (see also Kelso and Tuller, 1981 for a discussion of heterarchical and coalitional forms of brain organization): hierarchical systems induce a distinction between ascending and descending connections that could contribute to symmetry breaking which may then lead to directed coupling and information transfer (Tognoli and Kelso, 2014).

but not an elusive final destination, that really matters). This theory of information processing in the brain raises a question that may not have received enough attention: can we readily transpose findings from the smallest synaptic level—findings that speak of only a pair of neurons—to larger spatial scales such as neural areas or the whole brain? In all justice, countless emergent phenomena were discovered through this extrapolation, both in empirical and theoretical work (e.g., Sperry, 1969; Hopfield, 1982; McLeod et al., 1998). But it remains an uneasy feeling that so much of Brain Science is built upon the foundation of a pair of neurons, outside the context of their networks, and with two open-ended areas of darkness at either of their extremities that must be thought of as the entire remainder of the organism's brain (and body).<sup>2</sup>

#### **COLLECTIVE POWER OF NEURONAL SYNCHRONY**

We will come back to information transmission later, but let us now explore the matter of spatial scales. As humans tend to agree, increased size makes up for smarter brains (disclosure: both authors are human), and those bigger brains have room to organize themselves at multiple levels, coalescing into functional ensembles at several steps along the way up from neurons to functional areas and to the entire brain (Kelso, 1995; Bressler and Tognoli, 2006; Buzsáki, 2007; Kelso et al., 2013; Tognoli and Kelso, 2013). At larger and more integrated levels of description, other ordering phenomena were discovered that brain scientists conceive in terms of information exchange. In the late 1980's, two groups of scientists, one with Reinhard Eckhorn (Eckhorn et al., 1988) and another with Charles Gray and Wolf Singer (Gray and Singer, 1989), discovered that perceptual integration (or *Gestalt*) elicited transiently synchronous action potentials amongst neurons that had shared-stakes in the sensory object being viewed. Those neurons dealt with separate parts of the visual field, and they generally disagreed on when to elicit their action potentials in the regular course of their participation in visual function. Somehow however, through the complex labyrinth of the visual cortex and despite the fact that some finite amount of time was required to get from any one to any other of them (delays and frustrations manifested in their usual asynchrony), they managed to coincide when they responded to the same object. What we knew from those neurons is that they "responded" strongly to orientation, fragments of contours with sharp luminance gradients. Their synchrony it seems, was a trace of their joint participation in the construction of something bigger (the object) than what each of them was about (pieces of contour). These discoveries resonated with earlier theorizing regarding the organized behavior of neurons such as Donald Hebb's cell assemblies (Hebb, 1949) or Walter Freeman's mass action (Freeman, 1975). The findings by Eckhorn, Singer and Gray launched a relentless quest for synchrony in all parts of the brain and for numerous functions (von der Malsburg et al., 2010), and took the form of several variants (the most basic being coincidence of action potentials and phase-locking of neural oscillations).

### **IRRECONCILABLES**

Theories and dedicated experimental paradigms were built upon both discoveries of synaptic transmission and neural synchronization. And from each side, supporting evidence abounded (see previously mentioned references for evidence and reviews). In spite of their prominence and ubiquity though, the theories carefully avoided confrontation with each other, remaining mostly in the separate territories of distinct research groups.<sup>3</sup> One may note already some difficulties in reconciling them. Let us follow the two extreme views: perfect synchronization and perfect transfer. If all neurons were completely synchronized, they would remain in a changeless state of simultaneity. It is unclear how this system could have flows of information from one place to another. On the other end, if each neuron relayed information in a strict sense, the system would lack basic simultaneity through which synchronous phenomena could emerge. In their radical form it seems, the theories of information exchange *qua* synaptic transfer or neural synchrony are mutually exclusive.

#### **CAN WE FIND DIRECTIONS IN THE BRAIN?**

The tension is also visible in some empirical facts. Although directed flows of information in the Shannonian spirit do most certainly occur in neural networks, it is indeed quite challenging to track information otherwise than in local or statistical sense (by tracking, we mean to follow the path of information on a brain map as one would follow any object in motion on a symbolic representation of its spatial domain—see **Figure 1**). The

<sup>2</sup>Although much of neuroscience and its methodology focus on communication between pairs of neurons at the synaptic level, there is a long tradition of trying to understand the brain in terms of self-organization and dynamic coordination. For instance, Ross Ashby theorized on regulatory brain systems, considered under an equilibrium (homeostatic) perspective (Ashby, 1962) and he contributed to developing Cybernetics ("steermanship" born from control theory). From a more physical perspective, Arthur Iberall's Homeokinetics (formulated as Homeodynamics for biology) stressed the persistent, marginally stable nature of open thermodynamic systems which necessarily organize themselves as cyclic, physical action modes at all scales (see Yates, 2008, for review; also Eigen and Schuster, 1979 for a more formal but related approach). More recent years have seen the emergence of Synergetics as a foundational mathematical framework for nonequilibrium self-organization in physics, chemistry and biology (Haken, 1983). Synergetic self-organization is one of the cornerstones of Coordination Dynamics which stresses adaptive, informationally meaningful, bidirectional couplings on multiple levels (Kelso, 1995; Kelso and Haken, 1995; Kostrubiec et al., 2012; Kelso et al., 2013). A key notion of Synergetics that overcomes simple directional interactions between pairs of neurons is so-called circular or reciprocal causality (see discussions in Kelso, 1995; Kelso and Engstrøm, 2006; Tognoli and Kelso, 2013). In a multiscale perspective, circular causality manifests itself in terms of the micro and macroscopic states of a system reciprocally affecting each other; such that the macroscopic (ensemble) behavior enslaves the microscopic states, the ensemble average of which in turn produces the macroscopic states. It is possible that the formalism called for in this essay will rest upon outstanding developments along these lines. On the other hand, since "macro" and "micro" are relative terms in neurobiological dynamical systems where the timescales are not well-separated, a more systematic level by level approach may be required (see Kelso et al., 2013).

<sup>3</sup>There are few exceptions, for instance Fries' (2005) theory of communication through coherence that uses the phase of macroscopic patterns of coherence as a scaffold for selective modulation of information transfer at the microscopic level.

**FIGURE 1 | A teasing figure aimed at marking difference between directionality in well-formed Shannonian systems (as in this imaginary brain map) and in complex systems such as the brain**. In the latter, direction is less intuitive past immediate spatial and

temporal neighborhoods, and it can reverse across spatial scales of observation (Tognoli and Kelso, 2013). The question is highly relevant though, when one is concerned with where and how to effect changes in the system.

brain network after all, is a web, as Francisco Varela et al. (2001) emphasized, and one gets quickly lost with all the branchings, loops and loops within loops (Edelman, 1993; Gallos et al., 2012); structural features that "distribute" information (albeit unlike a postmaster distributes mail). So it seems that transmission principles do not scale well upward from simple "channels" of synaptic interactions to the larger and more complex web of evolved brains. Thus, it is without surprise that the brain betrays an essential communicational etiquette: its parts do not behave in a sequential—one-talks-at-a-time—manner (as opposed to the humoristic illustration of **Figure 1**). It is also overwhelmingly clear that "inputs" from the environment do not enter a silent system. Brain parts constantly exchange information about their current and past affairs, and what comes in at a given time works more as a "perturbation" to an already established ballet, an event that weaves itself within a broader scheme of coordinated brain behavior rather than the sole commander of all things present (Kelso, 1995). All of these nuances differentiate the brain from a channel in which information is transferred from sender to receiver. This situation creates mounting complications. The quest for directional flows in the brain has proved difficult both conceptually and methodologically, yet, it has not deterred efforts toward understanding. Mathematical and empirical studies aimed at resolving these questions are an active area pursued by many, including our own colleagues (Bressler et al., 2007; Bressler and Seth, 2011).

# **BRAIN ORGANIZATION: SYNCHRONIZATION OR COORDINATION?**

The second concept, synchrony, also bears its share of ambiguities. The firm ground on which we stand is that the timing of neural activity is not left to hazard (as if parts of the brain behaved independently, and were totally oblivious of what the others were doing). "*When"* one brain part behaves influences *when* others do. And like social creatures, neurons also use the power of their numbers to increase their impact, creating collective structures that speak from a common voice. A generic name for such behavior is *coordination* (Kelso, 1995). Synchrony is a narrower concept, one of several ways for a system to coordinate itself. Though synchrony has multiple meanings (and though its study uses a variety of tools across the board), it is easy to conceive and to model, perhaps explaining its systematic resort. To be rigorous however, synchrony requires two important and inter-related characteristics: first, that the underlying temporal order in which the system is embedded be frequency-locked, and second, that attractors have emerged in the system's coordination dynamics (attractors are mathematical structures that entrap the system's coordination dynamics into persistent—hard to break states). How to examine if there are attractors in the system from empirical data? We do know how to go from theoretical descriptions of dynamics with- and without-attractors to their phenomenology (using models to create data at will), but we are not very successful at taking the return path: demonstrating attractors or lack thereof from looking at data, at least for complicated systems like the brain. The other criterion, frequencylocking, is a little bit easier. And what the data say is that brain parts exhibit tendencies toward frequency-locking without going all the way to being perfectly identical. So it seems, we could be dealing with synchronous *tendencies* rather than synchrony (Kelso, 2008; Tognoli and Kelso, 2014). The difference may seem subtle to some (and some might be tempted to brush it aside as noise or measurement uncertainty), but mathematically and conceptually, it is enormous: it speaks of two entirely different species of dynamical systems, as said before, one with and the other without attractors. We are turning the spotlight to this distinction because we believe it to be paramount for progress in understanding the brain.

The mathematical formalism of "generalized synchronization" (Rulkov et al., 1995; see also Barreto et al., 2000) is apt to quantify such regimes of coordination sans attractors. Generalized synchronization assumes the existence of a functional mapping between two dynamical processes, without imposing the strong constraint of identical synchronization usually sought for in classical measures of coherence and correlation. The notion of generalized synchrony is set to grow in importance in the context of a coordination dynamic between weakly coupled components—a beneficial situation that enhances system complexity. The absence of complexity is a nefarious situation that André Gide related to tyranny.

# **AT THE CROSSROADS OF PROPAGATION AND SYNCHRONIZATION**

We hope that the previous exposé motivated the thought that neural networks neither operate on perfect synchrony nor on strict transfer, which is good news as each prevents expression of important features of the other. What then is the link between them? Some attempts at studying synchrony and transfer in a common formalism have emerged, exceptions to their usual avoidance of one another. One is the quest for quantifying directional coupling as discussed above. Another attempt is functional (relatedly, effective) connectivity,<sup>4</sup> a daring concept that Karl Friston created on his way to developing theoretical and computational tools for the analysis of functional images of the brain (Friston, 1994). Connectivity deals with ways for information to go from one place to another. Ideally, we would be able to measure the connection (the "traffic" between two sites) independently from the state of those sites where said traffic imparts effect (as one would measure how many cars travel on the road between two cities). If independently measured, large scale connectivity and local activity would be amenable to reveal their effect on each other. Since we do not have adequate tools to measure the flow of information in living fiber tracts at large though, connectivity is not measured directly; rather it is inferred from the way brain components behave. Interaction, it is postulated, has to leave detectable traces in the behavior of its participants. Of course, contemporaneous theories have shaped the lens through which scientists have tried to see this influence. To make things practical, the assumption was often made that regions exchanging information must be correlated or synchronized (connectivity→correlation). Flipping things around for the operational goal of quantifying the unquantifiable information flow, "how much regions were correlated" became the proxy for how much they exchanged information (correlation→connectivity). But with only this concept of synchronization under the scope, we may see a mere fraction of the brain at work, the tip of the iceberg. What if most coordinated behaviors in the brain do not fall under our definition of correlation or synchrony? Depending on the methods used, that would mean for instance brain regions that are coordinated yet not temporally coincident; or assemblies in which self-organization favors a fluid coordination regime sans-attractor (such as metastable tendencies, to be discussed below) over rigid states of phase-locking. Can we see dynamics in which no absolute "order" emerges in space (synchronization) or in time (transfer), and still make sense of it as a means for the brain to function? Those are the dark and uncharted areas in the spatiotemporal organization of complex systems—those for which we sorely lack concepts and methods (**Figure 2**).

#### **ENLARGING THE SCOPE: METASTABILITY**

The set of questions above resonates with a recent shift in perspective on brain function, from a primary focus on neural synchronization to the broader—and deeper—problem of dynamic coordination. This shift was salient in the editorial introduction to a special issue of Nature Review Neuroscience in February 2010, where the word "coordination" occurred six times in a short text of 250 words (From The Editors, 2010). And this is a concept that is growing (Kelso, 1995, 2012; Bressler and Kelso, 2001; Kelso and Tognoli, 2007; Werner, 2007; Rabinovich et al., 2008; Tognoli and Kelso, 2009, 2013, 2014; von der Malsburg et al., 2010; Farmer, 2011; Kelso et al., 2013). Coordination includes synchronization as one possible collective behavior, but it also considers many other ways for components of the brain to interact. In particular, under certain conditions partially synchronized behaviors arise. In them, the parts exhibit simultaneous tendencies to temporarily couple and to segregate as independent entities. Such metastable regimes, we and others have shown, constitute a recipe for complexity (Kelso, 1995, 2012; Friston, 1997; Freeman and Holmes, 2005; Kelso and Tognoli, 2007; Werner, 2007; Rabinovich et al., 2008; Tognoli and Kelso, 2009, 2013, 2014; Bhowmik and Shanahan,

<sup>4</sup>Karl Friston (1994) intended functional connectivity to expose temporal correlations between neurophysiological events—a descriptive approach without *a priori* specification that the originating structures would affect one another—whereas effective connectivity is meant to address explicit influences between brain parts (how much does A affect B, an estimation based on *a priori* model of those areas' relations, later reframed as causality).

sorely missing.

2013; Kelso et al., 2013). Why is metastable coordination dynamics of high interest to understanding how the brain works? First, it typically arises when the parts are no longer perfect clones of one another (e.g., as in computational models built from collections of identical neurons). When symmetry is broken and interacting parts are recognized in the diversity of their intrinsic behavior, a more realistic picture of brain function emerges (indeed a trend toward studying more diverse associations in the brain may explain the shift in perspective, for instance the interactions between neurons and astrocytes, see, e.g., Wade et al., 2013). Second, incomplete synchronization is more adaptive than pure forms. A fundamental nonlinearity in brain self-organization exists. Too much autonomy (parts of the brain hardly ever affected by what others are doing) prevents emergence, integration and mass action (Uhlhaas and Singer, 2006). Yet, too much integration (for instance the whole brain engaged in a giant common behavior, Cruikshank and Connors, 2008) is inadequate too, because the respective parts can no longer do what they are supposed to do in contributing to collective behavior. The parts then have no choice but to behave exactly like each other and the richness of their individual dispositions is lost to the ensemble. It is enough to note—as many have—that excess synchronization is pathological in the brain, for instance in epilepsy or Parkinson's disease (Uhlhaas and Singer, 2006; Hammond et al., 2007; Lehnertz et al., 2009). As a result, the ideal place for a brain to exhibit a rich set of meaningful behaviors is in-between integration and segregation. This is where the "incomplete" synchronization tendencies or metastable coordination comes into play (Tononi et al., 1994; Kelso, 1995, 2012; Friston, 1997; Werner, 2007; Chialvo, 2010; Tognoli and Kelso, 2013, 2014). Elsewhere, we have also speculated on the tremendous functional advantages that metastability would confer to a system, including speed, flexibility and resilience (Kelso and Tognoli, 2007; Kelso, 2012).

#### **CREATING A NEW CONCEPTUAL FRAMEWORK**

The gap between our current understanding of the brain and the miracles of our mental life and behavioral achievements (for example, consciousness and capacity for invention) remains abysmal. Looking through the history of science, several paradigms of brain function flourished and then dried up following the ebbs and flows of scientific metaphors. The ultimate model, the one that allows to forecast all matters of brain action and to design an artificial counterpart of multiple functional prowesses, remains out of sight. Two lines of thinking have been much explored in recent times: information transfer and synchronization. Their success owes much to the fact that they are special cases and open to quantification. When examined together though, they reveal some incompatibilities that seem to require a relaxation of both principles: less stringent temporal order and less complete spatial order. To advance our understanding of the brain, Neuroscience must open up avenues to study functional behavior in a broader sense. We face two alternatives: to leave it all within the current framework, with the approximate truth derived from current theories (the brain "sort of transfers information", and it operates with "near synchrony"), or to face the issue head on with a different theoretical mindset. In the latter case, a new phenomenology is up for grasp. It will be difficult to conceptualize, and even more so to observe, since it points toward a void in understanding. Modeling approaches can lead the way, by informing which observables we can expect to encounter in the coordinating living brain. And tools will have to be revised or built to adapt to this new world, tools that will say for instance, when "more synchrony" is "too much synchrony" (astonishingly, this simple question is not built into our current enquiries, despite obvious evidence of the ills of excess synchrony). We note that Brain Science is reaching a turning point that may make this renewal possible: it shows many signs of its readiness to enlarge the scope on brain function, not least of which is a recent outburst of interest in segregation phenomena (Poulet and Petersen, 2008; Ecker et al., 2010; Renart et al., 2010). A new paradigm would help to integrate principles that seem contradictory in their radical form: transfer and synchronization, as well as integration and segregation. Those pairs of concepts are reconciled under the dynamical regime of metastability (Kelso, 1995; Kelso and Engstrøm, 2006; Kelso and Tognoli, 2007; Tognoli and Kelso, 2014). Parts making-up the brain (for instance molecules, neurons, or brain areas) have simultaneous tendencies for independence and cooperation. As a result, they engage in the double-duty of lending their help to the collective behavior undertaken by several brain parts, and of performing their own independent behavior. Under a metastable regime, information is continuously created, preserved and annihilated by spatiotemporally changing coalitions among parts and processes. This is a source of dynamic complexity, and the likely origin of the human brain's many prowesses.

#### **ACKNOWLEDGMENTS**

This work was supported by NIMH (MH080838), NSF (BCS0826897), the US ONR (N00014-09-1-0527), the Davimos Family Endowment for Excellence in Science and the Chaire d'excellence Pierre de Fermat. Suggestions by both reviewers are gratefully acknowledged.

#### **REFERENCES**


system. *Proc. Natl. Acad. Sci. U S A* 91, 5033–5037. doi: 10.1073/pnas.91. 11.5033


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 20 March 2014; accepted: 09 June 2014; published online: 25 June 2014*. *Citation: Tognoli E and Kelso JAS (2014) Enlarging the scope: grasping brain complexity. Front. Syst. Neurosci. 8:122. doi: 10.3389/fnsys.2014.00122*

*This article was submitted to the journal Frontiers in Systems Neuroscience*.

*Copyright © 2014 Tognoli and Kelso. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

# Global segregation of cortical activity and metastable dynamics

#### Peter Stratton1, 2 \* and Janet Wiles <sup>3</sup>

<sup>1</sup> Queensland Brain Institute, The University of Queensland, Brisbane, QLD, Australia, <sup>2</sup> Centre for Clinical Research, The University of Queensland, Brisbane, QLD, Australia, <sup>3</sup> School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, QLD, Australia

Cortical activity exhibits persistent metastable dynamics. Assemblies of neurons transiently couple (integrate) and decouple (segregate) at multiple spatiotemporal scales; both integration and segregation are required to support metastability. Integration of distant brain regions can be achieved through long range excitatory projections, but the mechanism supporting long range segregation is not clear. We argue that the thalamocortical matrix connections, which project diffusely from the thalamus to the cortex and have long been thought to support cortical gain control, play an equally-important role in cortical segregation. We present a computational model of the diffuse thalamocortical loop, called the competitive cross-coupling (CXC) spiking network. Simulations of the model show how different levels of tonic input from the brainstem to the thalamus could control dynamical complexity in the cortex, directing transitions between sleep, wakefulness and high attention or vigilance. The model also explains how mutually-exclusive activity could arise across large portions of the cortex, such as between the default-mode and task-positive networks. It is robust to noise but does not require noise to autonomously generate metastability. We conclude that the long range segregation observed in brain activity and required for global metastable dynamics could be provided by the thalamocortical matrix, and is strongly modulated by brainstem input to the thalamus.

Keywords: thalamocortical matrix, autonomous metastable dynamics, cortical segregation, default mode network, sleep, wakefulness, complexity, spiking networks

# Introduction

In wakefulness and in rapid eye movement (REM) sleep, cortical activity exhibits persistent ongoing complex dynamics (Breakspear et al., 2003; Honey et al., 2007). In this state, activity shifts continuously throughout the cortex, and cortical regions couple (integrate) and decouple (segregate) across multiple spatial and temporal scales (Sporns et al., 2000a; Varela et al., 2001; Shanahan, 2008; Tognoli and Kelso, 2014). Each individual episode of integration and segregation is transient, but a continuous superposition of such episodes through space and time results in the observed persistent metastability of cortical dynamics (Tognoli and Kelso, 2014). This dynamical complexity is hypothesized to support the brain's flexibility and sophisticated processing capabilities (Breakspear et al., 2003; Buzsáki and Draguhn, 2004; Fries, 2005; Tognoli and Kelso, 2014), including memory retrieval, planning and problem solving (Binder et al., 1999; Mazoyer et al., 2001). During times when the brain is not actively processing sensory stimuli or task-related events, and as such is in a state known as the "resting" or "default-mode" state, cortical activity is

#### Edited by:

Emili Balaguer-Ballester, Bournemouth University, UK; Bernstein Center for Computational Neuroscience Heidelberg/Mannheim, Germany

#### Reviewed by:

Joachim Hass, Central Institute for Mental Health, Germany Loreen Hertäg, Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, Germany

#### \*Correspondence:

Peter Stratton, Queensland Brain Institute, The University of Queensland, QBI Building (#79), St Lucia, QLD 4072, Australia p.stratton1@uq.edu.au

> Received: 20 March 2015 Accepted: 07 August 2015 Published: 25 August 2015

#### Citation:

Stratton P and Wiles J (2015) Global segregation of cortical activity and metastable dynamics. Front. Syst. Neurosci. 9:119. doi: 10.3389/fnsys.2015.00119 concentrated in a well-defined sub-network including regions of frontal and association cortices (specifically the ventromedial prefrontal cortex, posterior cingulate cortex, ventral precuneus, and parts of the medial temporal and medial, lateral and inferior parietal cortices) (Greicius et al., 2003; Uddin et al., 2008). Activity of this default-mode network (DMN) is anticorrelated with activity in much of the rest of the cortex—that is, activation of the DMN and of those cortical centers used for sensory and task-related processing is largely mutually exclusive (Greicius et al., 2003; Uddin et al., 2008; Tomasi and Volkow, 2011).

Metastable cortical states, and the functions these dynamics presumably underpin (Binder et al., 1999; Mazoyer et al., 2001; Breakspear et al., 2003; Buzsáki and Draguhn, 2004; Fries, 2005; Tognoli and Kelso, 2014), cannot exist without the myriad, often overlooked, sub-cortical areas that provide the cortex with controlling and modulatory input. Projections from the pedunculopontine nucleus (PPN) and the laterodorsal tegmental nuclei (LDT), parts of the brainstem network collectively known as the ascending arousal system (AAS—but previously known as the reticular activating system), enter the intralaminar nuclei of the thalamus (IL) and thence on to the cortex through the thalamocortical matrix connections. The AAS is thought to modulate wake and sleep states as well as arousal and vigilance levels (Moruzzi and Magoun, 1949; Reese et al., 1995; Jones, 2003). The IL matrix connections project diffusely and somewhat non-specifically to large portions of the cortex, which in turn project back to the IL through the thalamic reticular nucleus (RN). Notably, these projections have opposing effects: whereas the cortex excites the RN, the RN exerts an inhibitory influence on the IL. Hence, rising global cortical activity increases RN activation, which in turn inhibits the IL, reducing its input to cortical activation and ultimately countering the activity rise in the cortex. Similarly, a decrease in cortical activity can cause an increase in thalamocortical input from the AAS. This diffuse matrix thalamocortical loop can therefore potentially dynamically control overall activity levels in the cortex (Steriade and McCarley, 1990). Effectively, it implements a mechanism similar to k-winner-take-all (WTA) across the entire cortex, where the allowed activity level k is controlled by AAS input to the IL. WTA networks are known to be able to implement powerful computational functions (Maass, 2000).

Brain integration is served by long range excitatory connections, but the paucity of long range inhibition in the brain has meant that the mechanisms to support long range segregation of brain activity are less obvious and are not well understood. Long range inhibition has been discovered within the visual cortex (McDonald and Burkhalter, 1993), between the hippocampus and entorhinal cortex (Melzer et al., 2012), and between the prefrontal cortex and nucleus accumbens (Lee et al., 2014). However, currently these long range inhibitory connections are known to exist only in or between a limited number of specific structures; they certainly do not approach the abundance of the long range excitatory projections forming the large fiber tracts that criss-cross the brain. The mechanism by which, for example, the DMN is segregated from other cortical regions is unclear (Greicius et al., 2003; Uddin et al., 2008). The contribution of the AAS, IL and RN to global cortical activation control is widely suspected (Steriade and McCarley, 1990; Taylor and Farrukh, 1996; Saper et al., 2001). In this paper we refine this view and argue for the specific functions of long range competition and segregation of cortical activity, and thereby the support of cortical metastability. To demonstrate this potential we present a computational model of the diffuse thalamocortical loop, called the Complex Cross Coupling (CXC) spiking network (see **Figure 1**). The CXC network includes input from the AAS and RN to the IL, local and long range corticocortical connections and local inhibitory interneurons. Simulations demonstrate that the model requires no extrinsic noise to exhibit its full range of dynamical states. We use the model to show how different levels of input from the AAS to the IL could support a range of dynamical states in the cortex, including states with high dynamical complexity.

# Methods

#### Neuron and Synapse Models

All simulations were conducted using the Parallel Circuit Simulator (PCSIM) (Pecevski et al., 2009), a comprehensive software package for the simulation of large neural networks. Simulations used Euler integration with a time-step of 0.1 ms.

The Complex Cross Coupling Spiking Network was constructed using two types of Izhikevich model cortical neurons (regular spiking, RS, and fast spiking, FS) (Izhikevich, 2003). These neurons provide for realistic neuron membrane dynamics such as spike frequency adaptation, intrinsic bursting, resonance and bistability, whilst being computationally tractable for large network simulations (Izhikevich, 2004). The Izhikevich model is defined by three equations over two variables, the membrane potential v (nominally in mV) and the membrane recovery variable q, which are updated as follows:

$$\nu' = \ 0.04\nu^2 + 5\nu + 140 - q + I \tag{1}$$

$$a' = \ a(bv - q) \tag{2}$$

if v ≥ 30 then

q

$$\begin{array}{c} \nu \leftarrow g \\ q \leftarrow q + h \end{array} \tag{3}$$

where I is the summed synaptic input current and a, b, g, and h are neuron-specific model parameters. In this study, all excitatory neurons were modeled as regular-spiking (RS) cells with the parameters (a, b, g, h) = (0.02, 0.2, −65, 8) and inhibitory neurons as fast spiking (FS) cells with (a, b, g, h) = (0.1, 0.2, −65, 2). These dimensionless parameters are set as specified in Izhikevich (2003) to obtain membrane dynamics (modeled by v) that closely resemble the modeled classes of neuron (i.e., RS and FS cells). In this study, pyramidal cells are modeled as regular spiking, and inhibitory interneurons as fast spiking.

The synapses received presynaptic spikes that would initiate postsynaptic currents that decayed with characteristic time constants. Synapses were conductance-based, meaning they each had a reversal potential Erev at which current flow ceased and beyond which the direction of current flow reversed. Reversal

potential for excitatory and inhibitory synapses was 0 mV and −90 mV respectively.

To implement short term synaptic dynamics (which was applied to synapses between all excitatory neurons in one experiment, the results of which are shown in **Figure 5D**), a combination of synaptic depression and facilitation was used (Markram et al., 1998). For synaptic depression, synaptic efficacy was assumed to be a finite resource, of which a proportion p was in use at any given time. Only an amount u of the currentlyavailable proportion of synaptic efficacy, (1 – p), was used at the occurrence of each spike; p recovered back to zero with time constant d (p was therefore bounded in [0,1)). The rate of change of p for synapse i, p˙<sup>i</sup> , was given by:

$$
\dot{p}\_i = 8(t - t\_i)(1 - p)\_i \mu\_i - \frac{p\_i}{d\_i} \tag{4}
$$

where δ is the Dirac delta function and t<sup>i</sup> was the time of the last spike from the neuron that was presynaptic to synapse i. The initial value of u for dynamic synapses was set to U, then for synaptic facilitation u was increased on the occurrence of each spike, recovering back to U with time constant f (u was therefore bounded in [U,1)). The rate of change of u for synapse i, u˙<sup>i</sup> , was given by:

$$
\dot{u}\_i = 8(t - t\_i)(1 - u\_i)U + \frac{(U - u\_i)}{f\_i} \tag{5}
$$

The synaptic usage factor U was set to 0.15 with a recovery time constant of f = 1000 ms for all dynamic synapses. Total synaptic current, Idynamic, for a postsynaptic neuron was given by the sum of all the currents from its afferent synapses:

$$I\_{dynamic} = \sum\_{i} \left[ w\_i (1 - p\_i) u\_i (\nu - E\_{rcv}) \right] \tag{6}$$

where w<sup>i</sup> was the total efficacy of synapse i and v was the postsynaptic membrane potential.

The synaptic recovery time constant d was set to 50 ms for local excitatory-to-excitatory and 500 ms for long-range excitatory-to-excitatory connections. The difference was motivated in part by the different recovery times for AMPA and NMDA receptors. Long range connections in this model were loosely associated with feedback connections within the hierarchically-organized, massively recurrent cortical connectome. These feedback connections may be NMDA-rich, as against feedforward and local connections which may be predominantly AMPA-based (Thiele, 2012). The time constants used were longer than typically acknowledged for these receptor types to compensate for the absence of other network influences in the model, such as cholinergic neuromodulation, which are known to enhance NMDAR function.

For static synapses (synapses of fixed efficacy, i.e., with no short term synaptic dynamics), synaptic efficacy was assumed to be an infinite (non-depleting) resource, so u was always 1 and the rate of change of p for synapse i, p˙<sup>i</sup> , simplified to:

$$
\dot{p}\_i = 8(t - t\_i) - \frac{p\_i}{d\_i} \tag{7}
$$

Total synaptic current, Istatic, for a postsynaptic neuron was then:

$$I\_{\text{static}} = \sum\_{i} \left[ \nu\_i (1 - p\_i)(\nu - E\_{\text{rev}}) \right] \tag{8}$$

Static synapses were used in all simulations in this study except for **Figure 5D** (for which dynamic synapses were used as above). For static excitatory-to-excitatory synapses the synaptic recovery time constant d was set to 50 ms, for excitatory-to-inhibitory 5 ms and for inhibitory-to-excitatory 40 ms. No inhibitory-toinhibitory connections existed.

#### Network

The CXC network was devised for this study in order to investigate the relationship between the diffuse thalamocortical matrix loop and cortical dynamics, although precursor models have been published (Stratton and Wiles, 2010a,b). The CXC model is an abstraction of the essential computational components of the complex thalamocortical connection structure. It utilizes some of the known characteristics of these regions (such as dense local and more sparse long-range connections), while ignoring others (such as cortical layers). As such, it is a general representation only, with no intended specific spatial scale. Its purpose is to show that thalamocortical-like connectivity between simple neuron-like elements can result in non-trivial dynamics, and to point to some of the general neuron and network properties that may be involved.

The network consisted of n<sup>e</sup> = 1000 regular-spiking (RS) excitatory pyramidal neurons and n<sup>i</sup> = 250 fast-spiking (FS) inhibitory interneurons connected linearly (i.e., in a 1-dimensional network) with directional synapses. Synaptic efficacies were set such that several presynaptic spikes in close succession were required to cause an output spike in a postsynaptic neuron. Connection structure was set to be small world-like, with dense local connectivity and sparse random long range connections, similar to the cortex. Each RS neuron was connected to each of its closest j neighbors with local excitatory efficacy wn/j where w<sup>n</sup> = 2 (the end neurons connected circularly to the opposite end of the neuron vector). Random long-range connections of weight wr/k where w<sup>r</sup> = w<sup>n</sup> were then made from each RS neuron to each other RS neuron with probability k/ne, giving an average of k long-range connections per neuron. In all simulations j = 4 and k = 10 (these numbers of connections were smaller than in actual cortex due to the limited size of the modeled network, and synaptic efficacies and currents were therefore scaled up accordingly). Inhibitory FS neurons were spread uniformly between the RS neurons. Each FS neuron received excitatory input from and projected inhibitory output to each of its closest l RS neurons with efficacy wi/l where w<sup>i</sup> = 1 and l = 20.

The thalamic reticular (RN) and intralaminar (IL) nuclei were each implemented as a single analog (non-spiking) neuron with output equal to the sum of its input currents. These nuclei are modeled as analog because their function is to balance cortical activity, and while the brain uses spiking neurons in these nuclei, we hypothesize that the summed action of many spikes from many neurons results in this homeostatic balance, so for the CXC model the analog representation is sufficient. RS neurons were innervated by IL projections with synaptic efficacy of 0.4. The IL received continuous steady (not time-varying) input current from the AAS (IAAS) which ranged from 0 to 10 in different experiments. This upper bound on IAAS was set to strongly depolarize the connected neurons without causing them to reach spiking threshold. Cortical neurons projected to the RN with efficacy 1, with synaptic currents decaying with time constant d, as above (Equation 7). The RN inhibited the IL with efficacy −1, completing the diffuse thalamocortical loop. Network activity was initiated by a single input pulse at time zero into a random selection of ne/2 RS neurons. Conduction delays were set in a uniform random distribution between 1 and 25 ms for long range connections and to 1 ms for all other connections.

To test the sensitivity of the results to numerical factors, additional test simulations were conducted with very high temporal resolution (1µs time-steps) and larger scales (100,000 neurons in the network). Additionally, simulations of 10.000 s duration were run to ensure that network dynamics neither failed nor entered a short limit cycle attractor state. All these simulations exhibited metastable network dynamics across their entire spatiotemporal extents.

#### Analysis

Input current from the AAS to the IL (IAAS) was varied from 0 to 10 in steps of 0.01. For each input current level, a simulation was conducted for 10 s of simulated time. Spike times and membrane potentials of all neurons were recorded at each time-step of 0.1 ms. The first 1 s of activity was deemed to be the network settling period during which time the network activity would transition from synchronous bursting (initiated by the input pulse) to sustained activity (or to quiescence depending on the input level). The neural activity for this first second was omitted from further analysis. The remaining 9 s of the recordings were used to calculate the mean firing rate, r, of the network (r = T/ne/9, where T is the total number of spikes generated by all RS neurons).

We estimated the local field potential (LFP) that would be recorded from this network at any moment in time, lfp(t), by summing the membrane potentials of all neurons:

$$\text{lfp}(t) = \sum\_{m} \nu\_{m}(t) \tag{9}$$

where vm(t) was the membrane potential of neuron m at time t. Due to the vertical alignment of cortical dendrites in real brains, real dendritic potentials, when summed from many cortical neurons, contribute to the LFP (Nunez and Srinivasan, 2006). Neurons in this study were simulated as point entities with no dendritic processes, so the neuron (soma) membrane potential was the closest estimate of the dendritic potential. This process of calculating the estimate of the LFP is similar to that used by Beim Graben and Kurths (2008).

An indication of the complexity of the dynamical network state was based on deviations of the interspike interval (ISI) distribution from exponential to multi-modal. During metastable dynamics, three modes (short, medium and long) were evident in the ISI distribution; these modes arose explicitly due to the processes of active integration and segregation in the network. Short ISIs were caused by bursty firing of a neuron, which occurred when the neuron was receiving strong synaptic input from neighboring neurons and from long range connections from distant neurons, and hence indicated that the neuron was being integrated into network activity. Long ISIs were caused by long periods when a neuron did not fire, which occurred when the neuron was receiving weak or no input from neighboring neurons and from long range connections, and hence when the neuron was segregated from network activity.

Spikes firing within the central ISI mode, midway between the short and long modes, were simply following the dominant oscillation frequency of the network, exhibiting neither enhanced integration nor segregation from network activity. Small integration and segregation peaks, when compared to the central peak, therefore indicated that most spikes were entrained to the dominant network oscillation and neuron firing was predominantly periodic, whereas simultaneously-large integration and segregation peaks indicated that the network was in a complex metastable dynamical regime.

Finally, the network trapping time was calculated (Marwan et al., 2002). Trapping time quantifies the amount of time a network remains in a given state before transitioning to a new state. The trapping time was determined by first constructing a matrix S of states of the network spiking activity over 0.2 s non-overlapping time windows. Within each time window j, the number of times each RS neuron i emitted a spike was counted:

$$S\_{i,j} = \left| \{ 0.2(j-1) \le t\_i < 0.2j \} \right| \text{ for all } i, j \tag{10}$$

where t<sup>i</sup> is the set of spike times of neuron i and |{...}|denotes the length of a set.

The correlation matrix of S was calculated and the resulting matrix was thresholded at 0.5, yielding the state recurrence matrix R:

$$R = H(\text{corr(S)} - 0.5) \tag{11}$$

where H is the Heaviside step function. R is a symmetric matrix that reveals, for all states j, which other states were similar (i.e., which other states had a similar pattern of firing neurons). Finally, the mean width of the super-threshold region surrounding the main diagonal of R was determined (i.e., for each 0.2 s window represented by an element on the main diagonal of R, the width of the super-threshold region around this element was measured perpendicular to the diagonal by stepping outwards from the element until a sub-threshold element, where Hi, <sup>j</sup> = 0, was found; the mean of these widths for all elements along the main diagonal of R is the trapping time of the network). Trapping time is interpreted as follows: If the network transitioned between states rapidly, then adjoining state vectors of S had sub-threshold similarity and the diagonal of the recurrence matrix R (the trapping time) was only one state wide (i.e., each state was similar only to itself). However, if the state transitions were slow, then adjoining state vectors of S had super-threshold similarity and the width around the diagonal of R was greater than one. Longer trapping times indicated that the network state was evolving more slowly. Trapping times approaching 10 s (the length of most simulations in this study) indicated that the network dynamics were fixed in an attractor state in which activity did not evolve at all.

## Results

# Autonomous (Self-sustained) Metastable Dynamics

To establish the baseline complex dynamics supported by the CXC network, constant input from the AAS was first set to the baseline level of 1 and synaptic conductances were set to standard values for the network (see Methods). No noise sources were used within the network or in its input. The network exhibited metastable dynamics (**Figure 2**) despite being deterministic. We have previously investigated the network properties required for a spiking network (a precursor to the CXC) to exhibit such dynamics (Stratton and Wiles, 2010a,b)—these include smallworld or scale-free cortical connectivity, a mechanism of global inhibition, and maintenance of dynamics in a critical (phase transition) state. The parameters of the CXC network of the current study were set to be in the critical region. Metastable network dynamics were robust to moderate network parameter changes, however significant modification of these parameters resulted in either collapse of network dynamics into a limit cycle resembling seizure or failure of activity to propagate causing the total network activity to fall to zero. These two states of seizure and quiescence are the low-complexity states between which the activity in the CXC network rapidly, partially and transiently switched to generate the metastability (Stratton and Wiles, 2010b) 1 . The cortico-thalamo-cortical feedback from the matrix connections played a homeostatic role, lowering cortical input as activity levels increased and raising input as activity levels decreased, ensuring that the network as a whole remained within the critical region of phase space. In all simulations, these autonomous, complex, non-periodic dynamics lasted indefinitely.

Inspection of the firing of neural assemblies (groups of neighboring neurons) in the spike raster plot shows that they occurred approximately 0.05 s apart (**Figure 2**). This oscillation period was determined by the interplay of the corticothalamo-cortical feedback and the time constant of the local synaptic inhibition. Despite the regular oscillation period, these assemblies formed at unpredictable, apparently random times and existed for unpredictable durations (**Figure 2**; also see Stratton and Wiles, 2010b). Dynamic formation of assemblies was unpredictable because it depended on three tightly coupled factors: (1) which assemblies were currently firing, (2) the precise pattern of random long-range connections from the currently active assemblies, since this pattern would determine which of the currently inactive neurons were receiving strongest synaptic input, and (3) the past history of firing activities, since neurons

<sup>1</sup>While the subcortical structure of the CXC is more complicated than that presented in this earlier paper, it fulfills the same function and the extra detail has been included for biological fidelity. Parameter sensitivity is therefore not significantly altered in the CXC.

neurons unpredictably forming transient oscillating assemblies, and ongoing non-periodic activity. (A) Spiking activity of 200 neurons for 1 s. Each horizontal group is an assembly of bursting neurons. Each neuron within an assembly bursts at high frequency, with the bursts occurring every 0.05–0.1 s (10–20 Hz) while the assembly is active. Assemblies are transient, sometimes firing synchronously with other assemblies in the network and

which had been recently active could still have been in a relative refractory period, meaning other neurons could fire first even if they were receiving weaker synaptic input. Thus, even though the network dynamics were noiseless and deterministic, they were unpredictable unless the entire network state (both static and dynamic, i.e., the entire connectivity map, the complete properties of every neuron and synapse, and the precise current states of all membrane potentials and synaptic currents) was known with absolute accuracy. In other words, the only way to predict the ongoing activity would be to create an exact duplicate of the network, and any inputs impinging upon it, and simulate it in its entirety. (Interestingly, this inability to accurately predict model dynamics for the CXC network applies equally to real brains).

#### Integration and Segregation

For the network with baseline AAS input level (equal to 1, as in **Figures 2**, **3A**), LFP power was strongest at approximately 19 Hz, indicating the dominant oscillation frequency in this network (**Figure 3A** center). The ISI distribution appeared monotonic on a linear scale (**Figure 3A** right inset), but by exploiting the fact that very small ISIs are excluded by the refractory period, we plotted the ISIs on a log scale to give superior count resolution for short ISIs (**Figure 3A** right). On a log scale several peaks were apparent; the ISI distribution was clearly multi-modal. The central mode, occurring at just below 0.1 s (–1 on the log scale), was caused by the dominant 10–20 Hz oscillations. This mode could therefore be considered the "base mode" or default ISI of this network. In this respect, spikes occurring 0.05–0.1 s apart formation and duration is evident. The time period shown in panel (A) is boxed. Mean firing rate for all neurons was just 1.0 Hz. (C) Spike raster plot for one typical neuron over 60 s, showing random firing with burst activity punctuating extended periods of quiescence. Most neurons had lengthy periods of complete silence lasting 10 s or more despite having mean firing rates around 1 Hz and the network as a whole oscillating at 10–20 Hz.

carried little information; these spike timings were predictable based on the observed dominant oscillation frequency, and in an information theoretic sense, the more predictable an event, the less information it conveys (Shannon et al., 1949). Unexpected, apparently random deviations from this base mode, however, can carry much larger amounts of information. Interestingly, for this network these deviations occurred in a manner suggestive of both integration and segregation of network elements, as follows:

The mode with shortest ISI, occurring at less than 0.01 s (– 2.4 on the log scale), was caused when neurons integrated into assemblies. When a neuron received synaptic input from its local neighbors and from random long-range connections from other active assemblies elsewhere in the network, it would fire at much higher rates than baseline and become integrated by and into the network activity.

Similarly, the mode with longest ISI in the ISI distribution (**Figure 3A**, right), occurring at approximately 1 s (0 on the log scale) and beyond, was caused by neurons firing less often than expected given the network oscillation frequency, or equivalently, neurons being excluded from firing. The majority of the principal neurons in the network had lengthy periods of complete silence lasting 10 s or more despite the network as a whole displaying regular oscillations of much shorter periods.

#### Input Level from the Ascending Arousal System Changed Dynamical State

When constant input from the AAS was increased, several dynamical properties of the network were altered (see **Figure 3B**,

#### FIGURE 3 | Continued

dominant 19 Hz oscillation (center). Firing rate variability was high; neural activity occurred in combinations of strong bursts and long periods of inactivity. The ISI distribution, shown as a probability density, was therefore multi-modal (right), indicative of integration (bursts), and segregation (extended periods of inactivity—see text). Displaying ISI with log time enhances resolution for short ISIs (main panel, right; inset on right shows the ISI distribution with linear time). Network parameters in this panel match those shown in Figure 2 above. (B) Increasing AAS input levels resulted in increased firing rate (left), higher dominant oscillation frequency (center) and a decrease in the number of spikes in the high and low modes of the ISI distribution (right) compared with the central mode, indicating that more

with four times the AAS input). The network firing rate increased as a direct result of the increased input allowing a greater number of neuron assemblies to be simultaneously active. The dominant network oscillation frequency also increased, due to the stronger input causing faster depolarization of the neurons after each oscillation cycle; the oscillation peak shifted from 19 to 26 Hz (**Figure 3B** center). In the ISI distribution, more spikes occurred in the central baseline mode and fewer in the outlying short and long ISI modes (**Figure 3B** right). This change in distribution indicated greater predictability of the spike train a greater proportion of spikes were being entrained to the overall dominant network rhythm and fewer were exhibiting enhanced integration with or segregation from this overall network activity. The reason for the reduced segregation is straightforward with more neurons firing there are fewer opportunities for neurons to be silent for extended periods. On the other hand, the comparatively lower integration into assemblies is due to stronger inhibition (which itself is due to the increased overall activity in the network); each oscillation cycle is shorter, so fewer spikes are generated by each neuron in each cycle before the assembly is silenced by inhibition.

For some small but non-zero levels of input from the AAS, the network entered a very different state (**Figure 3C**, AAS input lowered to 0.75). The dominant oscillation frequency decreased markedly and virtually all neurons were entrained to the global oscillation. This state had a very low complexity, as can be seen from the single large peak in the ISI distribution at the dominant frequency. In this state, the spike times of all the neurons verged on wholly predictable. It is not immediately evident why this state arose. However, two observations may be relevant: (1) the state was not strongly stable, with dynamics settling into either low or high complexity regimes under initial conditions with very small differences, and (2) the state was abolished by noise (see Section Robustness to Noise, below). These observations suggest that the network dynamics were bistable in this region of phase space (a not-uncommon occurrence in non-linear dynamical systems).

To establish a control condition with Poisson firing, which reveals the differences in dynamics when network activity is dependent on extrinsic noise rather than on the network structure, AAS input and all synaptic conductances in the network were set to zero efficacy and sufficient noise was added to the neural membranes to induce random spiking activity at 1 Hz (**Figure 3D**, left). The LFP showed a power spectrum spikes were becoming entrained to the dominant oscillation as AAS input increased. (C) Low but non-zero AAS input and no noise resulted in a very different dynamical state characterized by all neurons being entrained to a single low oscillation frequency (left). The LFP was characteristic of a rectangular wave (similar to a square wave but with a shortened or lengthened duty cycle) (center). Almost all ISIs were identical (right). (D) AAS input and all synaptic conductances in the network were set to zero and noise was added to induce random firing at 1 Hz (left). Mean firing rate was identical to that in (A) but the LFP was consistent with filtered noise, attenuated slightly at higher frequencies by the neural membranes (center). The ISI distribution was nominally exponential (inset, right) with the spike refractory period excluding short ISIs (main panel, right).

characteristic of low-pass filtered noise; higher frequencies were filtered by the membrane capacitance of each neuron (**Figure 3D**, center). The ISI distribution took the expected exponential shape curtailed by the spike refractory period. Poisson firing highlights the contrasts in dynamics between random activity and metastability, where the average firing rates were equal but the higher order statistics were dissimilar.

With the extrinsic noise removed and synaptic conductances reinstated (i.e., the network restored to standard baseline), AAS input was then varied from 0 to 10 in steps of 0.01, the mean firing rate for each instance was recorded, and the network trapping time was calculated (see Methods for details). Trapping time quantified the amount of time the network tended to remain in a given state before transitioning to a new state.

Mean firing rate increased linearly as AAS input was increased (**Figure 4**, top), except for low input values where firing rate was typically either zero or extremely high. The regions of zero network firing were generally for very low AAS input values near zero, while regions of very high firing rate occurred for slightly higher input levels between 0.6 and 0.8. The transformation between low and high firing rates was sudden, with no intermediate states. The dynamics in the high firing rate regions were as shown in **Figure 3C**—low complexity, global entrainment, and slow oscillations around 5 Hz. In these high firing rate regions the trapping time was high (**Figure 4**, bottom), signifying that the network state was not changing (marked as point **c** in **Figure 4**).

With increasing AAS input to just above 0.8, network dynamics again switched dramatically, in this instance to a state of low trapping time (high complexity) and low firing rate, as shown in **Figure 3A** (marked as point **a** in **Figure 4**). Trapping time and firing rate increased with further rises in AAS input (point **b** and beyond). The firing rate increase has been explained above. The trapping time increase is related to, and perhaps caused by, the increase in firing rate; specifically, by the increase in the number of simultaneously-active assemblies. When a greater number of assemblies are active at any given time, there are fewer long-range connections that project to currently-inactive assemblies. Only inactive assemblies have a chance of switching to an active state and changing the overall state of the network. The consequence is that, as firing rate increases, there is decreasing probability of a change in state at any given time, as reflected in the increasing trapping time.

#### Connectivity Impacts Dynamics

In all of the above simulations, long range cortical connections were set randomly with uniform probability. It is also possible to base connection probability on distance between the cells, so that cells that are nearer have a greater chance of having a long range connection between them. Such a connection scheme resulted in "waves" of activity propagating through the cortical cells (**Figure 5A**). These waves arose due to the connections from any active region in the network projecting most densely to regions that were immediately adjacent. These adjacent regions, receiving the strongest input, were the most likely to become active next. As this process repeated, the result was a wave of activity propagating through the network. However, since the propagation was chaotic, the speed, number and even the direction of the propagating waves could vary unexpectedly. Alternatively, by dividing cortical neurons into two groups such that intergroup long range connections were less likely than intragroup, activity would unpredictably switch between the groups (**Figure 5B**). The activity switch occurred for a similar reason to the activity waves in **Figure 5A**, except that the connection probability within each group was fixed and equal between all neurons, causing all neurons within a group to have equal chance to become active.

dynamics. The sporadic large changes in firing rate and trapping time for

Reducing the radius of local inhibition resulted in variable neuronal assembly sizes and much faster oscillations (**Figure 5C**). Assembly sizes varied because the reduced radius of inhibition was unable to contain the surrounding excitation, meaning sustained for all levels of AAS input down to zero (dashed line, top right). that adjacent assemblies could link together and form larger

super-assemblies. Oscillation frequency increased because local inhibition could no longer provide a stabilizing effect on the network dynamics; instead, local excitatory connections could form tight recurrent loops limited in frequency only by axonal conduction delays and membrane dynamics. Finally, increasing local inhibition and excitation strength and adding short-term plasticity (STP) to the excitatory synapses resulted in network dynamics which would spontaneously and intrinsically enter and exit seizure states (**Figure 5D**). STP seems to be involved in the generation of seizure-like dynamics in the model, since networks without STP do not exhibit these seizure states. Seizure onset was caused by chance synchronous firing, during one oscillation cycle, of a larger number of neurons than normal, simultaneously facilitating a large number of synapses, which pushed the network into a hyper-synchronous state that was then perpetuated (often beyond the facilitation time constant) by feedback through the thalamic matrix connections. Seizure termination also seemed to occur by chance, with a spike from a cell either not involved in the seizure or at a time not synchronized with the seizure oscillations. Such a "wayward" spike could sufficiently affect the timing of subsequent spikes in the network to break the hyper-synchronicity. However, a deeper understanding of the mechanism or mechanisms involved in seizure onset and termination would require further studies that are beyond the scope of the current paper.

In the version of the CXC network presented in this paper, the pedunculopontine, reticular and intralaminar nuclei are each modeled as unitary entities with no smaller components or sub-nuclei. While this simplification is adequate to demonstrate the functional involvement of these regions in long range segregation of cortical activity and facilitation of complex dynamics, in reality these nuclei are comprised of distinct components with distinct connection patterns. In particular, the "non-specific" matrix projections from the IL to the cortex may in fact have specific synaptic targets (Groenewegen and Berendse, 1994). This specificity has consequences for cortical dynamics: rather than cortical regions competing uniformly for activity, the competition (segregation) occurs in interconnected, hierarchically-arranged overlapping pockets of varying sizes across the entire cortex. With such a connection paradigm, total cortical activity remains controlled at all times, but inter-regional competition can vary in complex ways based on IL connection patterns and precise patterns of cortical activity at any given time. The result is that the repertoire of cortical dynamics is potentially even richer than presented here.

The network is not limited to linear (1-dimensional) layouts; 2-dimensional, and greater, network arrangements work equally as well (see online<sup>2</sup> for details). Overall, this and the above

<sup>2</sup>For an animated example of 2-dimensional cortical dynamics in the CXC see https://dl.dropboxusercontent.com/u/70159706/gtrajectory\_out.avi.

preliminary results (**Figure 5**) suggest a strong impact of network connectivity on the ensuing neural dynamics that warrants further investigation. Understanding how neural dynamics are influenced by specific connection structures in the brain such as the directed connections between regions at the macro scale, intra-regional connectivity inside structures like the hippocampus at the meso scale, and detailed neural circuitry at the micro scale, is critical to understanding how these structures perform their diverse functions.

### Robustness to Noise

To test the robustness of the network dynamics to noise, sufficient membrane noise was added to all principal cells to induce random spiking at 0.1 Hz in each cell; that is, spiking occurred at 100 Hz across the network of 1000 cells. (Note that this is a very different experiment to that depicted in **Figure 3D**, where the network was effectively disabled by setting synaptic efficacies to zero prior to the injection of noise.) In the case of adding noise to the functioning network, there was no significant change in the mean firing rate, integration and segregation, or the trapping time for most AAS input levels. However, for AAS input between 0 and 0.8, the transitions to low complexity states with high mean firing rates and high trapping times did not occur; instead, complex dynamics continued for all input levels down to zero (**Figure 4**, dashed line in top right panel). This continuation of complex dynamics was due to random spiking activity adding to the total activity in the network, similar to sustained higher AAS input, and holding the network in a metastable dynamical state. Crucially, the network's ability to maintain metastability and long range segregation of activity was not compromised by noise.

# Discussion

### Segregation, Metastability, and the Thalamocortical Matrix

The primary finding in the current study has been that the AAS-IL-RN circuit may have the ability to globally segregate cortical activity and maintain metastable cortical dynamics, with no need for injected noise or input perturbations, and that the global cortical state could be strongly modulated by brainstem input to the thalamus. In the absence of longrange inhibition, and the absence of long-range excitation onto local inhibitory interneurons, the long-range competition and network segregation observed in the CXC could only occur through the lowering of excitation from the IL. Local and longrange corticocortical connections in the CXC network were not strong enough to sustain activity at low firing rates—a depolarizing input from the IL was required to boost neural membrane potentials sufficiently close to threshold so that a small number of presynaptic spikes was capable of causing a spike in the postsynaptic neuron. If total activity in the CXC network increased, tonic depolarizing input from the IL decreased, and neurons or assemblies that would have fired due to convergent input from other active assemblies would then be unable to fire. This is competition through effective inhibition (inhibition that occurs via the withdrawal of tonic excitation), controlled by the total amount of activity across the network. Simultaneous, transient, unpredictable, recurring integration and segregation of activity in the CXC network resulted in the observed on-going metastability. Without these thalamocortical matrix connections, sustained metastability at low firing rates, long range competition, and global segregation of cortical activity would not be possible.

The activity states observed in the CXC network have analogs in real brains. The slow oscillations and global coupling seen at low AAS input levels in the CXC network are similar to the strong delta oscillations and very low dynamical complexity observed during deep (non-REM) sleep (Massimini et al., 2005; Murphy et al., 2009). In mammals, AAS input levels, particularly input from the pedunculopontine nucleus (PPN, or sometimes PPT or PPTg) are significantly reduced during periods of non-REM sleep (Moruzzi and Magoun, 1949; Reese et al., 1995; Jones, 2003), with slow delta oscillations at 0.5–4 Hz the predominant neural activity signature of this state. The transition between sleep and active cortical states is thought to be driven by cholinergic neuromodulation. However, there is likely to be more than one mechanism involved, and this result in the CXC network suggests that the transition could be assisted by the lowering of AAS input; supporting the switch from a state of high to low dynamical complexity.

At slightly higher AAS input levels than those required to generate sleep-like dynamics, the CXC network switches into a low firing rate, high complexity state. This state is analogous to awake states of low cortical arousal and metastability seen in real brains at intermediate levels of PPN activity, and observed during quiet relaxation (Stam et al., 1999). The characteristic neural activity signature of this state is relatively slow alpha oscillations (8–12 Hz, just lower than the dominant oscillation frequency observed in the CXC network) seen over large parts of the cortex. Increasing AAS input beyond this level in the CXC network resulted in increased firing rates, increased dominant oscillation frequency and increased trapping time—the length of time for which neural assemblies were stable—while decreasing the dynamical complexity. The decrease in complexity arose due to more spikes being entrained to the dominant oscillation and fewer spikes exhibiting either integration into assemblies or segregation from network activity. These states in the CXC network may correspond to states of attention, arousal and vigilance in real brains when AAS input is known to be maximal.

# Default Mode Network

Free association and daydreaming tend to occur when the brain is not actively processing sensory stimuli or task-related events, and as such these states are correlated with activity in the default mode network (Mason et al., 2007; Buckner et al., 2008; Christoff et al., 2009). It follows that when the brain switches state from free association to attention, activity tends to switch from the DMN to other parts of the cortex. The regions of the cortex involved in the DMN are more strongly connected to each other than they are to the rest of the cortex (Buckner et al., 2008). We have shown that when a similar connection strategy is employed in the CXC network—dividing the cortical neurons into groups where intergroup connections are less likely than intragroup—activity dynamically and unpredictably switches between groups.

In the CXC network, activity tends to concentrate in one group at a time because the recurrent long range connections required to sustain ongoing activity (Stratton and Wiles, 2010b) are focused within groups. Switching between groups is driven by a combination of the long range connections and the input from the AAS; when several long range inputs to a new group are activated simultaneously, perhaps coinciding with a decrease in activity of the currently-active group due to either habituation or random activity fluctuation, then input from the AAS can be sufficient to ignite a larger number of assemblies in the new group. Activity in the new group then competes for persistence with the current group through the segregation process mediated by the AAS and the thalamic matrix connections, and if activity in the new group is strong enough then it will dominate and an activity switch occurs. This dynamic switching in the CXC network provides a parsimonious explanation and inherent mechanism for activity in the DMN and other cortical regions being mutually exclusive despite the limited connectivity (especially inhibitory connectivity) between them. In the brain, the timing of switches in activity between the default mode and taskpositive networks is likely also influenced by task requirements and external events in the perceived environment, rather than being purely chaotic, but the segregation principle remains the same.

#### Consequences for Brain Function

Sustained firing of neural assemblies has been proposed as the neural substrate of working memory in the cortex (Wang, 1999; Pesaran et al., 2002; Jensen, 2006; Jensen et al., 2007). In the CXC network, the state of higher firing rates, increased trapping times and faster oscillations is analogous to a state of increased vigilance, attention and working memory in the cortex (Oken et al., 2006), driven by an increase of input from the AAS (specifically the PPN). There is ample evidence that increasing input from the AAS causes increasing cortical activation in general (Jones, 2003), and that firing rate and oscillation frequency also increase specifically for those neurons representing attended stimuli (Fries et al., 2001). There is also some evidence that increased vigilance does indeed reduce spiking variability (meaning that complexity of spiking patterns is also reduced) (Falkner et al., 2013). The converse state of low firing rates, short trapping times and high complexity in the CXC network is driven by lower AAS input levels and is associated with slower oscillations. This state is analogous to fluid states of mentation such as mind-wandering or daydreaming (Laufs et al., 2003; Mason et al., 2007; Buckner et al., 2008; Christoff et al., 2009) where working memory is not heavily utilized and mental associations arise freely and apparently randomly due to the increased variability of the neural dynamics.

The CXC network has a clear dichotomy between relaxed free association and vigilant attention; these modes cannot occur together because they are distinct dynamical states of the thalamocortical system, driven by changing input levels from the AAS. Dynamical states of low complexity are necessary for stable maintenance of working memory, with the trade-off that fewer potential states will be visited due to longer network trapping times. Dynamical states of high complexity and short trapping times are necessary for exploring more of the possible state space combinations of neural representations, supporting mind-wandering and free association, but are ineffective when focussed attention and working memory are required. Based on the model's behavior, we conjecture that states of high and low complexity are both useful but cannot co-occur, so the brain switches between them as need and opportunity arises.

Several predictions about neural dynamics can be made from the model:


What can the CXC network tell us about computation in the brain, rather than simply brain dynamics? To adequately address this question requires consideration of both representation (how does brain activity represent information?) and learning (how does this information come to be in the brain?); these are substantive and intricately connected topics which are the focus of much research today. In its current form, the CXC network offers mechanisms behind the intrinsic, autonomous generation of metastability that seem to be required for complex thought (Binder et al., 1999; Mazoyer et al., 2001; Breakspear et al., 2003; Buzsáki and Draguhn, 2004; Fries, 2005; Tognoli and Kelso, 2014) and for the stable maintenance of dynamic neural assemblies required for working memory (Wang, 1999; Pesaran et al., 2002; Jensen et al., 2007). These apparently conflicting requirements are addressed through the control exerted over network dynamics by the level of input from the AAS to the thalamus. With these mechanisms in place, the dynamics observed in the CXC can potentially inform existing and perhaps even all-new theories of information transmission and transformation in the brain. Likewise, theories of learning, representation and mental processing can be tested in, and may lead to refinements of, the CXC. Initial questions include:


Subsequent questions may address more complicated issues, such as how complex representations can be constructed in dynamical neural assemblies and utilized by the brain for purposeful computation. The ultimate goal of this future research will be to link the observed network dynamics with computational states in the brain. While realization of this goal is clearly distant, studying the CXC network will potentially lead in this direction.

#### Pathological Dynamics

Many brain disorders are associated with alterations to cortical dynamics, the DMN and the AAS (Garcia-Rill, 1997; Schnitzler and Gross, 2005; Uhlhaas and Singer, 2006; Buckner et al., 2008; Fröhlich et al., 2008; Fox and Greicius, 2010; Zhang and Raichle, 2010). We have demonstrated the dependence of autonomously-generated metastable dynamics on network connectivity in these regions. Because up to 99% of cortical connections derive from the cortex (Braitenberg and Schüz, 1998) and only a small fraction of the brain's energy consumption is used for the processing of external events (Raichle and Mintun, 2006; Zhang and Raichle, 2010), by far the majority of cortical activity originates in and is driven by other cortical activity. Changes in cortical structure driven by either learning or disease may therefore affect ongoing dynamics in unpredictable ways. Furthermore, these changes in dynamics may often lead to further structural change, creating a recursive dependency between dynamics and structure that could possibly lead to pathological states, such as in epilepsy, sleep disturbance, schizophrenia and many other brain disorders. By modeling autonomous metastable activity in the AAS-IL-RN circuit, this recursive chain of structural-dynamical co-dependence remains intact, allowing the investigation of how this dependence may lead to pathological states. Such studies are directions for future research with the CXC model.

#### Advances in the CXC Model

The thalamocortical matrix loop modeled in the CXC network is reminiscent of a winner-take-all (WTA) mechanism, however it is more accurately described as implementing winnerless competition (WLC). In WLC, a clear winning neuron or neural assembly never emerges from the competition; instead, each winner is immediately displaced by the next, resulting in continuously-evolving complex dynamic activity patterns (Akrami et al., 2012; Rabinovich et al., 2012). WLC can occur in networks of neurons mutually connected with inhibitory synapses, as observed in some simple animals during the generation of unpredictable behavior (Levi et al., 2005). It has also been shown analytically to occur in networks of neurons connected with slow global inhibition (Ermentrout, 1992). One of the insights offered by the CXC model is how the thalamocortical matrix can provide the necessary global inhibition to implement WLC for the control of complex dynamics across the cortex.

Random networks with balanced local excitation and inhibition have previously been shown to exhibit chaotic dynamics (Van Vreeswijk and Sompolinsky, 1996). Since then, studies have shown that networks connected using small world principles can also exhibit complex dynamics (Sporns et al., 2000b; Sporns and Tononi, 2001; Roxin et al., 2004; Riecke et al., 2007; Shanahan, 2008). However, for common topologies of these networks, the regions of parameter space where metastability was evident was small (Breakspear et al., 2003; Shanahan, 2008). More recently it has been shown that activity-dependent synaptic depression (a short-term decrease of synaptic efficacy based on postsynaptic activity) can massively enlarge the critical region where metastability occurs (Levina et al., 2007), and that the voltage-dependence of synaptic currents can stabilize complex dynamics for long periods of time (Kumar et al., 2008). Most recently, networks organized into hierarchical modules, where intra-module connections are abundant and inter-module connections are sparse, have been studied (Rubinov et al., 2011; Wang et al., 2011). Unlike previous networks, these latest networks (Kumar et al., 2008; Rubinov et al., 2011; Wang et al., 2011) can exhibit irregular sustained activity at low firing rates. For all of the above-mentioned networks, however, some or all of these questions remain open:


In the current study, we have shown how the thalamocortical matrix connections control cortical dynamics and resolve the above issues. Complex metastable dynamics in the CXC network continue indefinitely with no need for injected noise, at low firing rates and with very long ISIs occurring frequently. Most importantly, the matrix connections can cause explicit dynamic segregation of network activity through withdrawal of tonic excitation—a process we have termed effective inhibition. Effective inhibition leads to activity between weakly-connected regions (such as the default mode and task positive networks) being significantly anticorrelated rather than simply uncorrelated. A mechanism by which effective inhibition can arise globally across the cortex has not previously been suggested.

# Conclusion

The first and main result of this paper—that the global inhibition required for winner-take-all dynamics can be implemented by the diffuse thalamocortical loop—is not immediately obvious from neurophysiological observation, since there is little largescale long-range inhibition within the loop. For this reason we have termed the process 'effective inhibition'. Secondly, we show how changing tonic input levels from the ascending arousal system to the thalamus can change the dynamical state in the cortex (Section Input Level from the Reticular Activating System Changed Dynamical State). This result explains previous observations concerning how AAS input affects cortical activity (such as state changes between sleep, wakefulness and vigilance) and makes several novel predictions (Section Consequences for Brain Function). Thirdly, we show how cortical connectivity affects the sustained dynamics (Section Connectivity Impacts Dynamics). These sustained activity patterns can potentially be understood in terms of WTA dynamics, and we have demonstrated that the 'effective inhibition' paradigm, with its fundamentally different mechanism, is capable of supporting these patterns. When the network was structured into taskpositive and DMN cortical regions with dense local and sparse long-range connections, we additionally showed how activity could intrinsically alternate between the groups based on chaotic dynamics with no extrinsic noise. We argue that such intrinsic alternation provides a plausible explanation for the dynamical segregation of the DMN from other cortical regions. We have shown that the network is robust to noise, but importantly does not require noise for the generation and maintenance of a complex, ongoing brain state.

Previous studies have examined networks that exhibited complex dynamics, but these networks were unable to achieve

### References


long range segregation of activity. In contrast, the CXC network achieves both segregation of activity and metastability through the global control of the RN acting through IL. Without global control, network activity reduces to numerous interconnected pockets of activity that can mutually integrate due to activity propagating through the long range connections, but cannot mutually segregate.

## Acknowledgments

This work was supported by the Queensland Brain Institute, the Asia-Pacific Centre for Neuromodulation (a division of the University of Queensland Centre for Clinical Research), and an Australian Research Council (www.arc.gov.au) and National Health and Medical Research Council (www.nhmrc.gov.au) Thinking Systems Grant (Special Research Initiative: Thinking Systems; grant number: TS0669699). We thank our reviewers for their well-considered comments and suggestions which assisted us greatly.

visual attention. Science 291, 1560–1563. doi: 10.1126/science.10 55465


Jones, B. E. (2003). Arousal systems. Front. Biosci. 8, s438–s451. doi: 10.2741/1074


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Stratton and Wiles. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Extended Neural Metastability in an Embodied Model of Sensorimotor Coupling

Miguel Aguilera1, 2, 3 \*, Manuel G. Bedia1, 3 and Xabier E. Barandiaran4, 5

<sup>1</sup> Department of Computer Science and Systems Engineering, University of Zaragoza, Zaragoza, Spain, <sup>2</sup> Department of Psychology, University of the Balearic Islands, Palma de Mallorca, Spain, <sup>3</sup> ISAAC Lab, Aragon Institute of Engineering Research, University of Zaragoza, Zaragoza, Spain, <sup>4</sup> Department of Philosophy, University School of Social Work, University of the Basque Country (UPV/EHU), Vitoria-Gasteiz, Spain, <sup>5</sup> Department of Logic and Philosophy of Science, IAS-Research Center for Life, Mind and Society, University of the Basque Country (UPV/EHU), Donostia - San Sebastián, Spain

The hypothesis that brain organization is based on mechanisms of metastable synchronization in neural assemblies has been popularized during the last decades of neuroscientific research. Nevertheless, the role of body and environment for understanding the functioning of metastable assemblies is frequently dismissed. The main goal of this paper is to investigate the contribution of sensorimotor coupling to neural and behavioral metastability using a minimal computational model of plastic neural ensembles embedded in a robotic agent in a behavioral preference task. Our hypothesis is that, under some conditions, the metastability of the system is not restricted to the brain but extends to the system composed by the interaction of brain, body and environment. We test this idea, comparing an agent in continuous interaction with its environment in a task demanding behavioral flexibility with an equivalent model from the point of view of "internalist neuroscience." A statistical characterization of our model and tools from information theory allow us to show how (1) the bidirectional coupling between agent and environment brings the system closer to a regime of criticality and triggers the emergence of additional metastable states which are not found in the brain in isolation but extended to the whole system of sensorimotor interaction, (2) the synaptic plasticity of the agent is fundamental to sustain open structures in the neural controller of the agent flexibly engaging and disengaging different behavioral patterns that sustain sensorimotor metastable states, and (3) these extended metastable states emerge when the agent generates an asymmetrical circular loop of causal interaction with its environment, in which the agent responds to variability of the environment at fast timescales while acting over the environment at slow timescales, suggesting the constitution of the agent as an autonomous entity actively modulating its sensorimotor coupling with the world. We conclude with a reflection about how our results contribute in a more general way to current progress in neuroscientific research.

Keywords: neural assemblies, metastability, criticality, synaptic plasticity, embodied cognition, sensorimotor coupling, evolutionary robotics

#### Edited by:

Emili Balaguer-Ballester, Bournemouth University, UK

#### Reviewed by:

Paul Miller, Brandeis University, USA Heiko J. Luhmann, University of Mainz, Germany

> \*Correspondence: Miguel Aguilera sci@maguilera.net

Received: 23 May 2016 Accepted: 31 August 2016 Published: 23 September 2016

#### Citation:

Aguilera M, Bedia MG and Barandiaran XE (2016) Extended Neural Metastability in an Embodied Model of Sensorimotor Coupling. Front. Syst. Neurosci. 10:76. doi: 10.3389/fnsys.2016.00076

#### Aguilera et al. Extended Neural Metastability

# 1. INTRODUCTION

Generally, neurodynamic approaches have focused in understanding what kind of neural organization is necessary to cope with the requirements of an external world. Assuming that the brain is subject to demanding conditions from its environment, the challenge is to explain what type of neural computation or what form of organization of neural spatiotemporal patterns might be capable of satisfying the requirements for adaptive, conscious, cognitive activity. This has led to progress in the definition of a framework able to account for the brain's ability to display a rich set of meaningful behaviors. Nowadays, a popular view in neuroscience holds that the human brain is structured into a large number of areas in which information is highly segregated into local clusters and, at the same time, functionally integrated (Damasio, 1989; Varela, 1995; Tononi and Edelman, 1998). One of the most plausible mechanisms hypothesized to be behind this equilibrium between integration and segregation is metastable phase locking between neural assemblies over multiple frequency bands. This mechanism has been proposed to explain how the brain flexibly enters and exits coherent spatiotemporal patterns of neural activity (Kelso, 1995; Varela et al., 2001; Le Van Quyen, 2011). Subsequently, the notion of metastable neural assemblies as building blocks of brain organization has become relatively widespread in large-scale neuroscience studies (e.g., Werner, 2007a; Buzsáki, 2010; Edelman et al., 2011; Ward, 2011).

Nevertheless, when analysing and modeling brain organization, a crucial aspect of cognitive dynamics is frequently neglected: the sensorimotor coordination that continuously feeds back into brain dynamics (from saccadic eye movements to proprioception; from perception to action; O'Regan and Noë, 2001; Aguilera et al., 2013; Engel et al., 2013). Mental processes such as perception, emotion or intention are not limited to neural processes inside the brain, but produced through a flexible integration of the dynamics of brain, body and environment in a distributed manner. Hypotheses addressing this issue propose that brain organization consists in a plastic system of open loops developed in the process of life and closed to full functional cycles in every interaction with the environment (Fuchs, 2011), being the role of the central nervous system to transform and diversify these loops. In addition, it has been proposed that the behavior neural tissue in isolation might be restricted to little more than exhibiting spontaneous synchronization and other behaviors common to nonlinear dynamical systems, and the brain may operate as a metastable circuit breaker flexibly switching between different dynamic fields of agent-environment engagement (Dotov, 2014).

Furthermore, enactive approaches to neurodynamics have proposed that the formation and dissolution of neural assemblies in the brain must be embedded in sensorimotor regulatory cycles, producing the emergence of global organism-environment processes, which in turn affect their constituent elements (Thompson and Varela, 2001; Varela and Thompson, 2003; Di Paolo et al., 2016). One of the central contributions to this issue has been the notion of an operational closure of the nervous system (Varela, 1997; Di Paolo and Thompson, 2014), illustrated in **Figure 1**. Operational closure implies a circular regulation in which the coordinated activity of the neural system gives rise to the emergence of neural ensembles (or "cell assemblies"; Hebb, 1952), driving the behavior of the organism, which in turn generates a sensory input into the neural system closing a double regulatory loop. According to Francisco Varela,

The nervous system is organized by the operational closure of a network of reciprocally related modular subnetworks giving rise to ensembles of coherent activity such that: (i) they continuously mediate invariant patterns of sensorymotor correlation of the sensory and effector surfaces; (ii) give rise to a behavior for the total organism as a mobile unit in space. The operational closure of the nervous system then brings forth a specific mode of coherence, which is embedded in the organism. This coherence is a cognitive self: a unit of perception/motion in space, sensorymotor invariances mediated through the interneuron network (Varela, 1992, p.10).

Nevertheless, there are a lack of good models or precise characterizations of the operational closure of the nervous systems and the kind of interaction that takes place between neural, bodily and sensorimotor cycles (Barandiaran, 2016). Thus, it is far from clear how to characterize this sensorimotor specific form of coherence and how is it constitutive of cognitive activity. Clarifying this issue is of fundamental importance for embodied neurodynamic views in order to propose solid explanatory alternatives to internalist perspectives of brain organization.

In the case of metastability in cognitive processes, we know that metastable behavior is not restricted to the brain, but also extends to behavioral patterns (Kelso, 1995; Kelso et al., 1995). Nevertheless it is not clear what the relation is between behavioral metastability and underlying neural metastability. Consider, for example, the case of perceiving an ambiguous image (e.g., an image perceived either as a face or as meaningless

shapes, or a Necker cube which can be alternatively seen as oriented in two different positions) in which perception can alternatively and spontaneously switch from one mode to another. If we take a look at the brain we can observe the emergence of transient assemblies of neural synchronization when one mode of perception arises (Rodriguez et al., 1999), and at the same time if we were to analyse the switching of perceptive patterns the subject is engaged in we can observe signatures of metastable behavior (Kelso et al., 1995, Section 3). This raises questions around whether long term behavioral metastability is the product of a direct mapping of intrinsic metastable states of the brain into behavior or, conversely, metastability of both the brain and behavior is a product of the whole brain-bodyenvironment system coupled by sensorimotor processes (the interaction of neural dynamics, retinal activation, patterns of saccadic eye movements, etc.). These are difficult questions, requiring comparison between processes taking place at different scales (neurodynamic, sensorimotor, conductual, etc.) that pose important experimental difficulties. In the past, evolutionary robotics has been a highly productive tool for finding nonintuitive solutions to complex problems and understanding many-to-many relations between different scales of behavior (Nolfi and Floreano, 2000; Harvey et al., 2005). In the same tradition, in this paper we present an artificially evolved agent to explore the relation between neural and behavioral metastability in a simple bistable task. We choose a phototactic task<sup>1</sup> in which the agent alternatively develops a preference between two types of light (e.g., two different colors) as an example of a simple task involving metastable neural and behavioral dynamics.

Our hypothesis is that the slow modulation of synaptic plasticity over the sensorimotor coupling increases and sustains the metastability of neuro-behavioral integrated states in a manner that cannot be reduced to the dynamics of the brain in isolation, nor the brain receiving a structured input. These integrated metastable states are associated with specific modes of coherence in neural structures when they are engaged in bodily and environmental processes, as the brain-body-environment system becomes an operationally closed entity. The specificity of these modes of coherence is hypothesized to be related to the particularities of autonomous agency and the operational closure of the nervous system, i.e., the characteristics that allow us to describe an agent as an individual entity albeit in continuous interaction with its environment: such as the self-constitution of the agent as an entity or an asymmetrical interaction with its world in which it actively modulates its own sensorimotor coupling (see Barandiaran et al., 2009). Here, we propose a minimalistic approach to address some of the difficult questions arising from these ideas. We introduce a robotic model equipped with just three oscillatory units and synaptic plasticity in their connections. Because of its simplicity, this approach allows us to tackle the problem with a system about which we have complete knowledge and that is tractable using dynamical systems techniques of analysis.

In the following sections we first introduce the robotic agent and an artificial evolution process to obtain a model displaying metastable behavioral patterns in a bistable phototactic task. Then, we describe our methodology for analysing the role of the sensorimotor loop in the generation of metastable behavioral patterns, combining (1) the comparison of a situated agent interacting directly with its environment and a passively-coupled agent which is fed a signal identical to the one received by the situated agent, but it cannot influence its environment, and (2) a statistical description of the states of the agent and the environment together with the use of different tools from information theory to quantify the metastability in its behavior and the interaction between different scales of description of the robot (oscillatory activity, synaptic plasticity and behavioral patterns). In this framework we perform experiments showing that (1) the bidirectional coupling between agent and environment brings the system closer to a regime of criticality and triggers the emergence of additional metastable states, which are not found in the brain in isolation but extended to the whole system of sensorimotor interaction, (2) the synaptic plasticity of the agent is key to sustain open structures in the neural controller of the agent flexibly engaging and disengaging different behavioral patterns that sustain sensorimotor metastable states, and (3) this creates an asymmetrical circular loop of interaction between agent and environment, in which the agent is able to respond to variability of the environment at small timescales, while acting over the environment at large timescales. We conclude that metastability of neural dynamics can be extended to sensorimotor metastable states and that, in our model, this takes place when the agent establishes a specific circular relation with its environment, suggesting that the extension of metastable dynamics from the brain to interactive behavioral patterns is connected with specific forms of engagement with the world characteristic in autonomous agency.

# 2. METHODS

As we proposed above, our goal is to explore the relation between metastability in brain dynamics and behavior in a robotic model in order to test the hypothesis that some behavioral metastable states cannot be reduced to brain dynamics alone and are instead the product of an integration of brain, bodily and environmental dynamics. In order to do this, we design a model with the ability of presenting flexibility in both neural and behavioral patterns (which will be evolved using a genetic algorithm in order to reduce the constraints imposed onto the model) and we propose a framework of analysis allowing us to characterize metastable states and relations between components of the model, as well as a comparison of the behavior of the brain with and without the effect of the sensorimotor loop in equivalent conditions.

<sup>1</sup>Phototaxis implies climbing a light gradient. This task has been chosen because gradient climbing is a minimal task, which is widespread in nature. Many small scale adaptive behavior occurs along chemical gradients, and the microscopic world is full of gradients (like thermal gradients or light gradients but mostly chemical gradients). The adaptive behavior of small animals (e.g., C. elegans) and individual motile cells (e.g., bacteria but also animal cells migrating during development) is mostly a gradient-related adaptive behavior. Navigating smell or heat gradients are also stereotypical adaptive tasks for higher animals. Moreover many instances of higher-level behavior can also be interpreted as abstract gradient climbing (e.g., a human can move up a gradient of social popularity or economic wealth involving complex strategic decisions).

# 2.1. Model of a Neurodynamic Controller with Relational Homeostasis Embedded in a Robotic Agent

We propose a model of homeostatic adaptation inspired by previous work in evolutionary robotics (Iizuka and Di Paolo, 2007; Di Paolo and Iizuka, 2008), defining an adaptive mobile agent controlled by a plastic oscillatory neural system. This model is not intended to represent the activity of individual neurons but, more generally, to capture the dynamics of neural oscillations at a mesoscopic level, where integration mechanisms are hypothesized to be based on phase synchronization processes between neuronal groups (Varela et al., 2001), thus representing large-scale synchronization of brain regions that are anatomically far apart. Since the model is described in detail in Aguilera et al. (2015), we provide here a brief description.

The agent incorporates a neural controller defined as a fully connected Kuramoto network (Acebrón et al., 2005) with three units defined as:

$$\dot{\theta\_i} = \alpha\_i + I\_i + \sum\_{j=1}^{N} K\_{ij} \cdot \sin(\theta\_j - \theta\_i) \tag{1}$$

where θ<sup>i</sup> represents the phase of oscillator i, ω<sup>i</sup> is its natural frequency (range [0, 5]), Kij is the coupling strength between oscillators i and j, and I<sup>i</sup> represents the sensory inputs. The behavior of the neural controller is modulated by plastic mechanisms preserving phase relational invariances of the system, defined as:

$$\delta \dot{K}\_{\dot{i}\dot{j}} = \eta\_{\dot{i}\dot{j}} \cdot p(\Phi\_i - \Phi\_i^0) \cdot \sin((\theta\_{\dot{j}} - \theta\_i) - \Phi\_i^0) \tag{2}$$

where δKij are the connection weights, ηij is the rate of plastic change (range [0, 0.9]) of each connection, and 8 represents the phase difference of oscillator i with respect to the sum of the oscillators connected to it weighted by the strength of their connections:

$$\Phi\_i = \angle(\sum\_{j=1}^{N} K\_{ij} \cdot e^{\mathbf{i}(\theta\_j - \theta\_i)}) \tag{3}$$

where 6 denotes the phase of a complex value and i is the imaginary unit. 8<sup>0</sup> i (range [ <sup>−</sup><sup>π</sup> 2 , π 2 ]) stands for the preferred phase relation of the oscillatory node. Finally, the function p(x) determines the level of plastic change for all incoming weights of a node, which is activated when the value of 8i is far from 8<sup>0</sup> i (**Figure 2B**). When plastic changes take place, connection strengths change following a continuous nonmonotonic function Kij = α · F(δKij) (**Figure 2C**) designed to explore the full configuration space<sup>2</sup> , where α is a constant (range [0, 5]) that regulates the coupling strength.

In short, the model works under the assumption that largescale neural oscillatory components try to maintain a preferred phase relation with respect to other oscillatory components by means of plastically regulating the strength of their connectivity. The model is designed to present the possibility of metastable behavior at different states. Kuramoto oscillator networks can display metastable states when the connection strengths are below a critical point of complete synchronization. Also, we defined the evolution of synaptic plasticity in such a way that the

<sup>2</sup>This continuous, non-monotonic change of synaptic strengths was introduced by Iizuka and Di Paolo (2008) as a simple mechanism to avoid weight saturation of Hebbian-like rules. This way, eschewing biological plausibility allows us to make sure that the robots is able to explore its whole range of weight configurations.

FIGURE 2 | The robotic agent with three plastic oscillatory units. (A) Schema of the agent, the environment, sensors and motors, and the neural controller. (B) Plastic function <sup>p</sup>(8<sup>i</sup> <sup>−</sup> <sup>8</sup><sup>0</sup> i ), in which plasticity depends on the difference between the weighted phase relation 8i of the neural oscillator i with respect to other oscillators and the preferred weighted phase relation 8<sup>0</sup> i . (C) Mapping function F(δKij) which transforms weight values δKij into the actual value of coupling strengths between oscillators Kij. Reproduced from Aguilera et al. (2015).

agent can potentially explore all different available possibilities of engagement with the environment, and furthermore be able to exploit forms of behavioral metastability.

The agent is modeled as a simulated wheeled robot with a circular body of radius 4 and two diametrically opposed motors (**Figure 2A**), driving the agent backwards and forwards. The agent's mass is assumed to be small enough for inertial resistance to be negligible, thus its translational speed is calculated as the vectorial average of the motor velocities, and the angular speed as the difference of the motor velocities divided by the body diameter. Motor outputs are calculated from the phase relation 8<sup>3</sup> of the effector oscillator, with a gain parameter of value 2:

$$\begin{aligned} M\_l &= 2 \cdot \sin\left(\Phi\_3 - \Phi\_l\right) \\ M\_l &= 2 \cdot \sin\left(\Phi\_3 - \Phi\_l\right) \end{aligned} \tag{4}$$

where 8<sup>r</sup> and 8<sup>l</sup> (range [0, 2π]) are bias terms which map the motor output into the actual motor activation.

The agent has two pairs of sensors (right and left) for each of the different light sources A and B. Each sensor points to a direction at π/3 radians from the forward direction. Light A sensors are connected to oscillator 1 and light B sensors are connected to oscillator 2. The effects of both the angle and the distance on the sensor activation are represented by the following function:

$$I\text{DX} = \begin{cases} \frac{s\_{DX} \cdot 0.5 \cdot (1 + \cos(\alpha\_{DX}))}{1 + e^{\alpha \cdot (d\_{DX} - b)}}, & |\alpha\_{DX}| \le \pi/2\\ 0, & |\alpha\_{DX}| > \pi/2 \end{cases} \tag{5}$$

where X can represent either light A or B, D stands for either right or left sensor, αDX is the angle of sensor DX to light X, dDX is the distance between sensor DX and light X, and a and b have the arbitrary values of 0.03 and 100 respectively. The light intensity received at each sensor is multiplied by a gain parameter sDX (range [−8, 8]), feeding the resulting value to the corresponding oscillator's input I<sup>i</sup> . A full schema of the robot is represented in **Figure 2A**.

All parameter values (except where otherwise specified) are fixed by a genetic algorithm within the indicated range. A population of 20 agents is evolved using a rank-based genetic algorithm with elitism. Each of the agent parameters ωi,sDX, α, ηij, 8r, 8<sup>l</sup> and 8<sup>0</sup> i is encoded into a 5 bits string representing real numbers uniformly distributed within the specified range. For each generation, the best 4 agents (20% of the population) pass to the next generation without change. For the remaining slots, pairs of individuals are selected for crossover with a probability proportional to their fitness value, and new individuals are created mixing their genes (bit series) by adding a mutation probability of 3% for each gene.

The agents are evolved for displaying a metastable behavior in which the agent has to develop switching preferences toward two different types of light. That is, there is an environment with two types of light (e.g., two colors) and we want the agent to develop a preference toward one of them (e.g., repeatedly interacting with it) while being able to switch its preference to the other light depending on its internal configuration. This behavior is chosen because it demands the agent to present a robust phototactic behavior while at the same time presenting flexibility in the creation and dissolution of behavioral preferences. An evaluation procedure is proposed for the genetic algorithm in order to accomplish this objective, consisting in four different tasks designed by Iizuka and Di Paolo (2007): a single light A, a single light B, one light A and a blinking light B, one light B and a blinking light A (blinking lights illuminate with a probability of 0.15 for each time step). The agent gains fitness by approaching the non-blinking light. The objective of this configuration is to create a "dummy" that encourages the agent to learn to ignore one of the lights while approaching the other. Lights appear at a random distance, [100, 150]. When two lights are present, they appear, from the agent's point of view, with a random separation within the range [π/2, 3π/2]. The length of each trial is 125 s.

Each individual agent is tested for 12 independent runs (3 for each of the 4 tasks) consisting of a series of trials where a light or a pair of lights are presented to the agents for a fixed time. Synaptic weights δKij are reset to initial random values before each run. Each run consist of 8 trials in which the agent is presented with one or two lights for a specified time. Only the last 3 trials of each run are evaluated in order not to penalize slow plastic changes and bootstrap evolution. All simulations are run with an Euler step of 0.1.

Fitness for each trial is calculated in three terms, Ftrial = (F<sup>D</sup> + Fp) · FH. F<sup>D</sup> = 1 − d<sup>f</sup> /d<sup>i</sup> , where d<sup>f</sup> and d<sup>i</sup> respectively correspond to the final and initial distances to the target light. F<sup>p</sup> is equal to the proportion of time that the agent spends within a distance of less than 4 times its body radius (i.e., a distance of 16) to the target light during a trial. F<sup>H</sup> represents the mean level of homeostasis in the system, computing the mean degree of homeostasis <sup>1</sup> 3 P i (1 − p(8<sup>i</sup> − 8<sup>0</sup> i )) (i.e., 1 minus the level of plasticity) for each oscillator. The genetic algorithm is run for 500 generations, reaching a stable level of fitness around 0.45. The best performing agent from the last generation (which is able to reach both lights ignoring the dummy) is selected. The code simulating the behavior of the agent and the parameters obtained from the genetic algorithm can be accessed from the following repository https://github.com/IsaacLab/HNA-roboticmodel/tree/master/minimal-preference-task.

# 2.2. Conceptual Setup for Testing Sensorimotor Integration: Situated vs. Passively-Coupled Agents in An Open Environment

In order to explore how the agent exploits internal and sensorimotor metastability we simulate the agent in an open environment (which was never experienced during training) in which the agent can develop sustained preferences toward the two types of light. In this case, the two lights are always present with equal intensities, and a new pair of lights is generated periodically after a given time (starting a new trial). The best performing agent from the last generation of the genetic algorithm agent is simulated in a virtual environment which presents a series of pairs of lights, giving the agent a time of 1250 steps to choose and approach one of them. As analysed elsewhere (Aguilera et al., 2015), the agent is able to develop stable preferences toward one of the lights, maintaining it for several trials until the preference is changed. The switching of preferences depends on the long-term interaction between internal plastic mechanisms and the encountered configurations of the environment. Different neural cell assemblies arise connected with particular patterns of behavior of the agent, and at slower timescales synaptic plasticity modulates the emergence and dissolution of these behavioral patterns. A video of the behavior of the agent (including plastic mechanisms) can be found at https://vimeo.com/53847420.

Once defined the agent in which we want to explore the emergence of metastable behavioral patterns, we propose a sensorimotor null model to be compared with our model in order to test the influence of the sensorimotor loop in the generation of metastable states in the agent. To this end, we propose an agent which maintains the structure of the received input but presents a disrupted coordination with its environment. Thus, we will compare:

1. A situated agent with normal sensorimotor interaction.

2. A passively-coupled agent, in which the input fed to the agent is recorded from the behavior of the situated agent. Thus, in the passively-coupled agent the received input is decoupled from the activation of the motors but it maintains an structure as if it was generated by real interaction.

With this comparison we can detect the effects that, despite being the result of sensorimotor coordination, cannot be replicated just by using an input with an adequate structure. This is a subtle difference, but if genuine sensorimotor coordination is constitutive of a cognitive process, the same process should not take place when the agent is passively processing an input with the rig as an input-structured process.

# 2.3. Discretization and Probability Density Function of the System

In the experimental setup defined above, we will use information theory tools in order to get a better understanding of how the different elements of the neural controller and the agent's behavior interact, using a symbolic representation of the system states. Understanding the coordination between neural ensembles and sensorimotor activity is not trivial due to the moderately high dimensionality of the system (9 dimensions of the neural controller, plus the dimensions of body and environment). Nevertheless, the system can be simplified by reducing both the state of neural ensembles and synapse configurations to discrete values representing the state of a network, plus a binary variable representing the behavior of the agent (reaching one light or the other).

To simplify the analysis, we are interested in a description of the system minimizing the number of symbols needed to describe the states of the robot, while maintaining the properties of the system. In this case, we find that a binary discretization is a good choice for describing the system. In order to ascertain a good discretization of the system, we test the validity of different possible discretizations by comparing the dimensionality of the original and discretized data through principal component analysis on their covariance matrix. We use the following definition of dimensionality (Abbott et al., 2009):

$$d = \left(\sum\_{i=1}^{N} \widetilde{\lambda}\_i^{\;2}\right)^{-1} \tag{6}$$

where <sup>λ</sup>e<sup>i</sup> are the normalized eigenvalues of the covariance matrix expressing the fraction of the variance explained by the corresponding principal component. We find that different discretizations generally increase the dimensionality of the system (due to the introduction of noise in the form of discretization error). In **Figure 3** we can observe how the dimensionality of the discretized systems departs from the original dimensionality depending on the number of bins employed. Specifically, we find that the choice of a binary description with just two bins is a particularly good description of the covariances of the system, increasing the dimensionality of the system by just 1.32% of the original dimensionality. Therefore, for the rest of the paper we will employ a binary description of the variables of the system as described below.

In previous work (Aguilera et al., 2015) we have determined that the configuration of the oscillators in the assembly is the relevant variable for generating one or other type of behavior. Therefore, we define cell assemblies depending on the relative phase of the oscillators with respect to the mean phase of the Kuramoto network. We codify the state of the cell assembly with a string of three bits 2:

$$\Theta = \{\Theta\_1, \Theta\_2, \Theta\_3\}, \text{ where } \Theta\_i = \begin{cases} 1, & \text{if } \sin\left(\theta\_i - \theta\_m\right) > 0 \\ 0, & \text{otherwise} \end{cases} \tag{7}$$

where θ<sup>m</sup> = <sup>6</sup> ( 1 3 P i e <sup>i</sup>θi) is the mean phase of the system at a particular instant.

Analogously to cell assemblies of neurons, we may consider a constellation of changing synaptic weights as an assembly

FIGURE 3 | Increment of dimensionality of the discretized system for different bin choices. The dimensionality of the data generated by the situated agent (the passively-coupled agent yields similar results) with continuous (dashed line) or discrete (continuous line) values using different numbers of equally spaced bins in the discretization. We observe how a binary description (2 bins) is a good discretization of the system, presenting a small increase in dimensionality only matched by discretizations with 7 or more bins. A discretization with 3 bins is particularly unfit since it cannot capture the covariance of small fluctuations around the central bin.

of synapses or "synapsemble" (Buzsáki, 2010, p. 372). Synapse assemblies have been hypothesized to be critical for building up and dissolving cell assemblies and linking together sequences of cell assemblies. To codify the activity of synapse assemblies, we define each synapse as active or inactive if the value of the synapse is higher<sup>3</sup> than the mean value of the synaptic strengths Km, where K<sup>m</sup> = 1 6 1 T P t P i,j Ki,j(t). The state of the synaptic assembly

9 is codified with a string of 6 bits:

$$\Psi = \{\Psi\_{1,2}, \Psi\_{2,1}, \Psi\_{1,3}, \Psi\_{3,1}, \Psi\_{2,3}, \Psi\_{3,2}\}, \text{ where}$$

$$\Psi\_{i,j} = \begin{cases} 1, & \text{if } K\_{ij} > \langle K\_m \rangle \\ 0, & \text{otherwise} \end{cases} \tag{8}$$

Similarly, for each trial we define a variable 3 which represents the behavioral pattern of the agent (i.e., what light it reaches):

$$\Lambda = \begin{cases} 1, & \text{if } d\_{f,A} < d\_{f,B} \\ 0, & \text{otherwise} \end{cases} \tag{9}$$

where df,<sup>A</sup> and df,<sup>B</sup> are the final distances to each type of light at the end of the trial.

In summary, we define whether a particular cell or synapse ensemble is active by using a set of binary variabless = {2, 9, 3} which represent if a specific ensemble is active at a particular moment of time and what behavioral pattern is being developed by the agent. All possible relations between oscillators give rise to 6 possible states for the cell assemblies, and all possible activated synapses give rise to 64 possible combinations or synapse assemblies, giving us a complete and discrete definition of the system that we can use to apply information theory tools. In order to do so 100 similar agents with random initial conditions are simulated for the situated and passively-coupled cases for 1000 trials with a duration of 1250 steps, generating a time series with 1.25 · 10<sup>6</sup> states. Sections 3.2 and 3.3 use one of these time series to compute mutual information through time and transfer entropy, although the other 99 series yield practically identical results. The calculations in Section 3.1 require us to accurately compute the whole probability density function of the system. Thus, in order to avoid correlations in our sampling we compose 100 series of 10<sup>6</sup> states, consisting in 10<sup>4</sup> random states extracted from each of the initial 100 time series. Each sample can provide an estimation of the frequency of the 2<sup>10</sup> states of s, inferring the probability density function of the system, which we compute as P(s) = ns/nT, where n<sup>s</sup> is the number of occurrences of state s, n<sup>T</sup> is the total length of the sample.

### 2.4. Information Theory Tools

Having defined these variables of the system in a discrete manner, we can use information theory tools to determine the relation between variables. In our model, we can use these measures to quantify the relation between the state of different elements in the neural controller of the agent, or between such components and features of the environment surrounding the agent. The information contained in a random variable is quantified in terms of entropy, which is defined as:

$$H(X) = -\sum\_{\mathbf{x} \in X} P(\mathbf{x}) \log(P(\mathbf{x})) \tag{10}$$

where X is the set of states of the variable and P(X) its density probability function.

A useful measure to compare two variables is the relative entropy or Kullback-Leibler divergence between their statistical distributions, which is a measure of the difference between two probability distributions X and Y. It is defined as:

$$D(X;Y) = \sum\_{\mathbf{x}\in X} \sum\_{\mathbf{y}\in Y} P(\mathbf{x}) \log \frac{P(\mathbf{x})}{P(\mathbf{y})} \tag{11}$$

Given a pair of variables X, Y and their marginal distributions the Kullback-Leibler divergence can be used to capture the information shared between the variables, defined as their mutual information:

$$I(X;Y) = H(Y) - H(Y|X) = \sum\_{\mathbf{x} \in X} \sum\_{\mathbf{y} \in Y} P(\mathbf{x}, \mathbf{y}) \log \frac{P(\mathbf{x}, \mathbf{y})}{P(\mathbf{x})P(\mathbf{y})} \tag{12}$$

By definition, I(X; Y) = I(Y; X), thus mutual information cannot describe relations of causality. Instead, transfer entropy measures are typically employed to analyse causal relationships between variables. The decrease of uncertainty in the state of a variable derived from the past history of other variables is defined as the transfer entropy between two variables:

$$TE(X \to Y) = H(Y\_{t+\tau} | Y\_t^{(d')}) - H(Y\_{t+\tau} | Y\_t^{(d')}, X\_t^{(d)}) = \sum\_{\substack{\mathbf{x}\_{t+\tau}, \mathbf{x}\_t \in X}} \sum\_{\mathbf{y}\_t \in Y} P(\mathbf{x}\_{t+\tau}, \mathbf{x}\_t^{(d)}, \mathbf{y}\_t^{(d')}) \log \frac{P(\mathbf{x}\_{t+\tau}, \mathbf{x}\_t^{(d)}, \mathbf{y}\_t^{(d')}) P(\mathbf{x}\_t^{(d)})}{P(\mathbf{x}\_{t+\tau}, \mathbf{x}\_t^{(d)}) P(\mathbf{x}\_t^{(d)}, \mathbf{y}\_t^{(d')})} \tag{13}$$

where X (d) t denotes the past history of x counted from time t and length d (i.e., xt, xt−1, ..., xt−d).

### 3. RESULTS

In this section we present results comparing a situated and a passively-coupled agent using a discrete description of the system and information theory tools. We first show that when the system is coupled to its environment it is brought closer to a regime of criticality and that the number of metastable states of the system is extended. These extended metastable states do not arise from the brain nor the agent in isolation but from the whole brain-body-environment system. We then use tools of mutual information to observe how neural plasticity in the agent in coordination is key for generating the structures that sustain flexible and metastable behavioral patterns. Finally, we use transfer entropy to characterize the loop of interactions between oscillatory dynamics, synaptic plasticity and behavioral patterns, describing the circular multiscale relation between the agent's neural controller and its behavioral dynamics necessary for generating extended sensorimotor metastable states.

<sup>3</sup>K<sup>m</sup> is an arbitrary threshold for capturing when weights are active and sufficiently strong. Different values have been tested without altering the results.

# 3.1. Scale-free Statistical Distribution and Mestastability

We start by analysing the properties of the statistical distribution of the situated and passively-coupled agents. We compute the probability density function P(s) for each agent, where s = {2, 9, 3}, by sampling 100 simulations of identical agents with random initial conditions during 1000 trials (1.25 · 10<sup>6</sup> steps). We generate 100 samples, each one composed of 10<sup>4</sup> random statessrandomly sampled from each simulation run (10<sup>6</sup> states in total) and calculate the frequency of occurrence of each state. The result from one randomly selected sample is shown in **Figures 4**, **5**, while in the text we provide the statistics of the complete set of 100 samples.

An initial finding about the probability density function of the agent is that it approximately follows the Zipf law (**Figure 4A**), for states with a probability larger than 2 · 10−<sup>4</sup> . If the occurrence of the states of the system s is ordered by their decreasing frequency P(s), Zipf's law states that P(s) decays as the inverse of their rank r(s) in the ordered sequence, making P(s) ∝ 1/r(s). The occurrence of Zipf-like distributions is considered to be a signature of criticality (Mora and Bialek, 2011), coinciding with previous analysis of self-organized critical patterns in the same robotic agent when it is coupled with its environment (Aguilera et al., 2015). We observe that, while the situated agent presents a pattern very close to Zipf's distribution, the passively-coupled agent diverges more from a perfect scalefree distribution (**Figure 4B**), especially in states with higher probabilities. We can quantify this divergence by computing the Kullback-Leibler divergence between the distribution of states of the agent P(s) and Zipf's law distribution Psf(s) ∝ 1/r(s). As we observe in **Figure 4C** the divergence from the Zipf distribution in the passively-coupled agent is more than twice that of the situated agent. Computing the average and standard deviation for the 100 generated samples reveals that this result is repeated for different agents, confirming that the Kullback-Leibler divergence to a Zipf distribution is much larger in the passively-coupled agent (µ = 0.381, σ = 8.44 · 10−<sup>4</sup> ) than in the situated case (µ = 0.141, σ = 5.46 · 10−<sup>4</sup> ).

Criticality in the brain is generally associated with the metastability of transiently formed neuronal assemblies (Werner, 2007b), although in general the exact relation between the existence of metastable states and criticality is still not well understood. The definition of a metastable state is a state whose energy is lower than any of its adjacent states while not being the state of minimum energy of the system. If we assume that the probability of each state follows a Boltzmann distribution<sup>4</sup> , metastable states will be those with higher probabilities than those of their adjacent states. We define adjacency between two states when they are separated by a single flip<sup>5</sup> of an individual variable 2i, 9i,<sup>j</sup> or 3. In short, we consider metastable states as local peaks in the probability landscape. If we compute the number of metastable states of the system s = {2, 9, 3}, we observe that the situated agent presents 17 metastable states for most of our data series (µ = 17.2, σ = 0.782)), while the passively coupled agent presents typically over 13 metastable states (µ = 13.1, σ = 0.902), indicating that the level of metastability is boosted when the agent is in interaction with its environment. However, if we only analyse the neural system of the agent, i.e., the system s ′ = {2, 9}, we find that the situated agent presents 13 metastable states (µ = 13, σ = 0) and the passively-coupled agent presents around 11 (µ = 11.0, σ = 0.243). **Figure 5** portrays the number of metastable states of one random sample.

These results show how the critical scaling and the repertoire of metastable states of the agent is extended when it is coupled with its environment. Moreover, metastable states generated when coupled with the environment cannot be reduced to metastable states in the "brain" of the agent (i.e., in s ′ = {2, 9}) but only appear when we analyse the distribution of the complete system (s = {2, 9, 3}). These results suggest that, aside from neural metastable states generated by oscillatory dynamics and neural plasticity, sensorimotor metastable states can appear from the coordination between behavioral patterns and internal

<sup>5</sup>Other definitions of metastability could be used, using an arbitrary number n of flips. Nevertheless, in this particular case only n = 1 yields the existence of mestastable states.

<sup>4</sup>The situated system can be considered an isolated system in thermodynamic terms, whereas the passively-coupled system may be considered a closed system exchanging energy but not matter with the situated system, thus making plausible the assumption of thermal equilibrium and a Boltzmann distribution of states, and therefore a direct mapping between probabilities and the energy of each state.

neural dynamics. Spin glass theory indicates that metastable states emerge when some of the couplings between variables are negative (Mezard et al., 1987), which can be translated to stating that in our case agent-environment effective coupling presents mechanisms of mutual inhibition between pairs of variables. The appearance of metastable states only existing for the situated case in the whole sensorimotor system suggests a complex regulation between neural and sensorimotor processes, inviting us to take a closer loop of how agent-environment relations take place in order to increase the metastability of the system.

# 3.2. Mutual Information Flows

Typically, features such as criticality and metastability have been linked to the idea of systems driven by interaction dynamics between its components (Jensen, 1998; Van Orden et al., 2003; Ihlen and Vereijken, 2010). We investigate what type of interaction takes place in our agent between oscillatory dynamics, neural plasticity and behavioral patterns to generate neural and sensorimotor metastable states. We use information theory tools to quantify the interactions between the components of the system in the situated and passively-coupled agents, simulated for 1000 trials with a duration of 1250 steps. Although we only show the results of the time series of an individual simulation (random sampling is not applicable if we want to maintain temporal correlations), the differences in the results analyzing different runs of the simulation were negligible.

First, we analyse what information is shared by the emergent cell assemblies 2, synapse assemblies 9 and behavioral patterns of the agent 3 by measuring mutual information along the time series of values of each variable. In **Figure 6** we can observe how the three variables share an important amount of information. The entropy of 3 (which is the variable with the lowest entropy) is 0.86, thus the shared entropy is in the same order of magnitude in most cases. In the case of the situated agent, we can observe in **Figure 6** (left) that all variables share a relevant amount of information. However, in **Figure 6** (right) we observe that the information shared between 9 and 3 decreases dramatically, suggesting that most of the interaction between the two variables is lost.

The information analysis above shows a static picture of information flows on average, but it does not explain how these flows unfold over time. To overcome this limitation, Beer and Williams (2015) have proposed a framework combining information flow and dynamical analyses, exploring how a simulated model agent in a relational categorization task integrates information at different moments of time about a cue used for solving the task. Instead of analysing information as an average of the dependences between variables along a time series, they run the same task several times for different initial conditions and compute information measures for each time instant. Instead of using a series of temporal values of a variable, they use a series of values of a variable on each instant along different starting conditions. Similarly, we fold our time series into 1000 time series (one for each simulated trial) in which e.g., 2′ T (t) = 2(t + (T − 1)1250), where T is the trial number and 1250 is the duration of each trial. Consequently, for each value of t we have 1000 values of 2′ T (t) we can use for computing mutual information with other variables. For six consecutive trials (i.e., t = 1, ..., tend = 6 · 1250) we compute the mutual information between the neurodynamic variables of the agent 2′ T (t) and 9′ T (t) and the behavioral pattern at the end of the sixth trial 3′ T (tend) (in order to observe how information about future behavioral patterns is accumulated). Also, we compute the joint mutual information that both 2′ T (t) and 9′ T (t) share with 3′ T (tend).

In **Figure 7** (left) we can observe the result for the situated agent. We see how the mutual information I(2′ T (t); 3′ T (tend)) increases during the middle of the trial, fading out at the beginning and the end. That is, the activation of specific neural patterns when the robot is approaching a light contributes to its repetition in future trials (i.e., contributing to the habit of choosing that light). However, when the robot engages in other behavior (e.g., exploring its surroundings, or stopping after having reached a light) the habit is no longer being enacted and the information about it vanishes from its oscillatory patterns. Also, from one trial to the next the information at the peak increases, being maximal at the sixth trial. We can interpret this as a self-sustaining behavior of cell assemblies: when a cell assembly emerges, it reinforces itself and has more probabilities to reemerge in the next trial. Similarly, I(9′ T (t); 3′ T (tend)) steadily increases until a cell assembly is activated at the middle of the sixth trial. Mutual information between 9′ T (t) and 3′ T (tend) is continuously accumulated and does not decrease, thus we hypothesize that the configuration of the synapse ensembles "stores" information about the behavior that the agent will develop. Furthermore, when we analyse the joint mutual information I(2′ T (t), 9′ T (t); 3′ T (tend)), we observe that it is always higher that the individual contributions. Moreover, agent (left) and passively-coupled agent (right).

it increases when I(2′ T (t); 3′ T (tend)) decreases at the transitions between one trial and another. Also, I(2′ T (t), 9′ T (t); 3′ T (tend)) decreases when at the middle of the trial a cell assembly is activated, except in the last trial. This portrays an interesting picture, where information flows back and forth between the emergent cell assemblies and the collective cell-synapse assemblies, until the sixth trial when an assembly emerges producing behavior 3′ T (tend).

If we analyse the passively-coupled agent we observe a quite different picture (**Figure 7**, right). Although I(2′ T (t); 3′ T (tend)) is quite similar in both cases (its values are slightly smaller in the passively-coupled condition), I(9′ T (t); 3′ T (tend)) does not integrate any information. This suggests that even when the input produced by behavior 3′ T (tend) is able to influence the cell assemblies that emerge, coordination between behavior and the stabilization of synapse assemblies does not take place. Furthermore, the joint information I(2′ T (t), 9′ T (t); 3′ T (tend)) does not integrate much information either, and the anticorrelation between I(2′ T (t); 3′ T (tend)) and I(2′ T (t), 9′ T (t); 3′ T (tend)) disappears. This suggests that the passively-coupled agent does not capture the struggle between information flows through individual and collective variables, indicating that the important moments for generating the behavior of the agent are not only synchronizing moments of emergence of cell assemblies, but that most information is built during instants of desynchronization corresponding to transition from one assembly to the next.

These results show that synaptic plasticity, in coordination with behavioral patterns, plays a fundamental role in the situated agent, since it allows the agent to "store" information about future behavioral patterns the agent will engage it. Interestingly, the results in **Figure 7** suggest that neural assemblies behind the execution of a specific behavioral pattern reinforce the synaptic circuits sustaining that pattern, which store information about the repetition of that behavioral pattern. This resonates with the idea of the brain as a plastic system of open loops (Fuchs, 2011) created during previous interactions with the environment and functionally closed to full sensorimotor cycles in every new coupling with the environment. Sensorimotor metastable states could be precisely the transient closure of those loops, which in turn imprint and reinforce onto the brain the synaptic structures necessary to their reproduction.

# 3.3. Transfer Entropy

The analysis above shows the information shared by variables unfolding through time. As mutual information is a symmetric index, it is not a good tool to characterize causal interactions between parts of a system. Instead, we characterize directional interactions by measuring transfer entropy between variables using Equation 13, with d = d ′ = 1 as the length of the past history that we take into account<sup>6</sup> and a logarithmically distributed series of values of τ from 1 to 625, 000 steps (half the length of the 1000 trials), with multiplicative intervals of 100.1 . In **Figure 8** (left) we can observe a complex chart of information flows for the situated agent:


timescales, the flow does not exist in the opposite direction (**Figure 8**, middle-left), suggesting that the behavioral pattern of the agent influences the cell assembly that emerges, but that the current cell assembly that is active at a particular moment of time is not decisive for the behavioral pattern that the agent will deploy.

• 9 − 3 transfer. There is an important bidirectional exchange of information between 9 and 3. This suggests that 9 is the variable that determines the behavior that will be chosen by the agent. Also, we can observe that TE3→9 and TE2→9 are very similar in value and shape (if we integrate the area of the difference between TE3→9 and TE2→9 and divide it by TE2→9 the result is 0.11, showing that both functions coincide with almost 90% of accuracy). This is supported by the fact that there is no functional dependency from 3 to 9, since change in the weights δK˙ is only a function of K and θ. As we can easily check, all the other information flows present in **Figure 8** correspond to actual functional dependencies depicted by the equations defining the systems<sup>7</sup> . This suggests (since 3 influences 2 and not otherwise), that 3 causally determines 2 which in turn influences 9 in their circular mutual interaction<sup>8</sup> .

<sup>7</sup> Since θ˙ is a function of θ, K and sDX, and s˙DX is a function of θ, K and sDX. The only functional dependency that presents no informational content is TE2→3 8 It is worthy noting that in the experiments depicted in **Figure 8**, the variable 3 is defined as the macroscopic behavior of the agent, whereas 2 and 9 are patterns of low level variables of the agent. To test that this difference did not distort in any way the result of our analysis, we repeated the analysis substituting 3 by a variable

that the duration of a trial corresponds to a value of τ of 1250 steps.

<sup>6</sup>We tested different values of d and d ′ up to 5 for a series of representative values of τ with similar results, therefore we used d = d ′ = 1 to reduce the computational cost, since just varying the value of τ seems to be enough for capture the different timescales of information flows.

Putting together the transfer entropy flows above, we may summarize them in the schema at **Figure 9** (left). The behavior of the agent 3 generates an input that determines the emergence of cell assemblies 2 at fast timescales. A circular relation between the emergent cell assemblies and their underlying synapse ensembles 9 generates a particular behavior 3 which is determined by the state of 9 at longer timescales. We can observe how the resulting schema is similar to the one proposed by Varela (1997) and depicted in **Figure 1**, though adding interesting information about the timescales of each dependency.

Moreover, for the passively-coupled agent, the information flows from 9 to 3 and from 9 to 2 are disrupted and reduced in comparison to other flows, whereas information flows from 3 to 2 and 2 to 9<sup>9</sup> are maintained or even increased. This disruption of part of the transfer entropy flows dissolves the self-sustained neurodynamic structures that generate a coordinated behavior and reinforcing unidirectional influence from 3 to 2 and 2 to 9, as we depict at **Figure 9** (right).

These results strongly suggest that the generation of complex and integrated neurodynamic structures is a product of a double circular loop that strongly couples (1) neural oscillatory patterns with the plastic synaptic structures sustaining them and (2) these neurodynamic circular structures with the behavioral patterns generating them. This double loop is constituted in a way that creates a circular asymmetry between agent and environment, in which the oscillatory dynamics of the agent present a sensitivity of environmental parameters at shorter timescales, and its

<sup>9</sup>Flows from 2 to 9 are almost equivalent to flows from 3 to 9, supporting our suspicion that they represent the same information flow.

repeated activation generates the synaptic structures that are able to engage and disengage different behavioral patterns of the agent at longer timescales. The operational closure of the nervous system implies a special circular relationship between agent and environment, in which the agent individuates itself in front of its environment as it is capable of being sensitive to small fluctuations of its world while being able to act over it at longer timescales as a coherent dynamical unit.

# 4. DISCUSSION

In this paper we have presented a neurodynamical model of oscillatory activity with synaptic activity embedded in a robotic agent in a behavioral preference task. The model is based on a network of three Kuramoto oscillators with plastic homeostatic mechanisms designed to maintain constant phase relations among oscillators. Our goal was to explore metastability in behavior and neural assemblies in a context of embodied, adaptive activity, in which the agent is in continuous and bidirectional interaction with an environment, a dimension which is frequently neglected in the study of brain activity and organization. The model was carefully designed for exploring (1) the integration of transient assemblies underlying behavior through nonlinear coupling neural clusters generating specific conducts in the agent, and (2) the coordination between sensorimotor and plastic neurodynamic structures into a selfmaintaining behavioral patterns. The integration of these two levels of activity gives rise to metastable sensorimotor integrated patterns which cannot be reduced to metastability of brain dynamics alone, as behavioral preferences of the agent emerge from the interaction between oscillator cell ensembles, ensembles of synaptic weights and the agent's sensorimotor coupling. We present a methodological framework to analyse the role of sensorimotor behavior in interaction with neural dynamics: we compare a situated agent, normally interacting with its

Frontiers in Systems Neuroscience | www.frontiersin.org September 2016 | Volume 10 | Article 76 |

S defined as a string of 4 bits (one for each sensor of the robot), in which each bit was equal to one if the activity of its corresponding sensor is higher than its average activation. The results of transfer entropy measures were strikingly similar to those displayed in **Figure 8**, indicating that the series of sensory activation are strongly correlated with particular behavioral macroscopic patterns of the agent. Since the use of 3 simplifies other analyses performed in this paper, we have chosen to use this variable instead of S.

environment, and a passively-coupled agent, receiving a sensory input recorded from the situated agent but unable to influence its environment in any way. Comparing a passively-coupled agent and a situated agent in an open environment we have found three different results shedding light on the relation between neural and behavioral metastability.

These results are obtained first, through a statistical description of the agent (in the form of a discrete characterization of the states of the agent neural configurations and behavioral patterns), depicting how the situated agent presents signatures of criticality and additional metastable states that are not present in the passively-coupled agent. Moreover, we show how those additional metastable states do not appear in neural variables alone, but in the combined space of neural and behavioral patterns, indicating that metastable states in behavior are not exclusively a direct mapping of neural metastable states. Instead, we find the existence of sensorimotor metastable states that extend the range of metastability of the agent's "brain."

Second, an analysis of the flow of mutual information between different groups of variables of the agent shows that in both the situated and passively-coupled models neural ensembles of oscillatory components contain a lot of information about the behavioral pattern being developed. However, this information is lost after the neural ensemble dissolves at the end of a trial, and only in the situated case this information is stored instead through changes in synaptic plasticity, generating the structures reinforcing future appearances of that behavioral pattern. This suggests that mechanisms of synaptic plasticity have a fundamental role in coordinating neural and behavioral processes, sculpting the sensorimotor structures sustaining extended metastable states. While synapse assemblies have been hypothesized to have a critical role for building up and dissolving metastable cell assemblies in the brain as well as linking together sequences of cell assemblies (Buzsáki, 2010, p.372), our results invite us to rethink the role of synapse assemblies in a broader sense, as fundamental elements that facilitate emergence and dissolution of metastable modes of engagement with the environment.

Finally, using transfer entropy we can depict the causal influences at different timescales between different components of the system. We observe how the situated agent generates a closed network of interaction, circularly organized with bidirectional interactions at different timescales. This network takes the form of a double coupling loop of (1) a circular causal dependence between the emergence of cell assemblies and the synaptic neural structure that generates them, and (2) a causal chain in which synaptic structures influence the behavior displayed by the agent, which in turn triggers the emergence of specific neural assemblies (**Figure 9**, left). If the sensorimotor loop is disrupted (e.g., when the agent is passively-coupled, and probably for more severe sensorimotor disruptions), this circular closure disappears and autonomous organization of the agent vanishes (**Figure 9**, right). This provides cues of what happens in real-life examples of disruption of sensorimotor coordination. For example, in Held and Hein's experiment on visuallyguided behavior (Held and Hein, 1963), the "passive" kitten fails to develop perceptual abilities. Similar situations can take place in different physiological or pathophysiological conditions. Examples of those are the problems faced by deafferented subjects (i.e., without any proprioception) to develop behavioral automatisms exclusively in the absence of sensory feedback; needing to rely on visual feedback to perform simple tasks such as holding an egg without breaking it, or when are unable to maintain an upright posture in the dark (Cole and Paillard, 1995).

Interestingly, the schema of circular dependencies in the agent resembles the idea of operational closure of the nervous system proposed by Varela (1997) depicted in **Figure 1**. Moreover, it describes a novel characteristic of closure since this loop of closure creates a multiscale asymmetry between agent and environment: the agent is sensitive to changes in the environment at fast timescales, while it can influence the environment at slow timescales (**Figure 8**, left). Although the agent's oscillatory dynamics are mostly driven by inputs from the environment at fast timescales, it exerts an influence over the environment at the slower timescales of synaptic plasticity by generating structured behavioral habits (e.g., reaching repeatedly one of the lights) which will influence future stimuli received by the agent. In some sense, once the agent "sees" a light it is trapped in a behavioral field and has to reach it, but it still has a degree of autonomy in the sense that it can modulate its internal connectivity to influence which lights it is going to be sensitive to in the future. This allows us to identify the agent as a unit which is affected by bottom-up causal flows of sensorimotor stimuli and, at the same time, it is able to develop a downward causation modulating its sensorimotor interaction. Breaking the symmetry of the coupling between agent and environment has been proposed as one of the fundamental aspects which can constitute an agent as an autonomous entity able to regulate from within its exchange with the world, constituting its identity as a self-individuating system (Barandiaran et al., 2009). Previous characterizations of agent-environment asymmetry have referred only to the presence of a directionality in the flows of information from agent to environment depicting a causal influence from the former to the latter (see Seth, 2007; Bertschinger et al., 2008). These contributions do not take into account a selfreferential operational closure of the system (Bertschinger et al., 2008, p.14) and only quantify the degree of self-determination of the system. In contrast, our approach captures agentenvironment asymmetry as a circular relation of causal influences at different scales.

The type of analysis performed in our model has typically been unexplored by neuroscientists studying real life organisms, partly due to the difficulty of recording whole-brain activity of freely behaving animals. In general, recordings of neural activity have been limited to either small brain regions or to immobilized or anesthetized animals exhibiting limited behavior. Nonetheless, during the last few years some promising results point to the plausability of experiments involving the sensorimotor engagement of whole-brain or large brain areas. For example, an interesting technique for analysing brain and behavioral activity of a head-restrained mouse interacting with a virtual reality environment in a spherical treadmill (see Dombeck et al., 2007) has been developed. Furthermore, the first report of whole-brain recording in freely behaving animals has been reported for the nematode Caenorhabditis elegans during free locomotion (Nguyen et al., 2016). These advances open-up an exciting path for neuroscience, allowing an exploration of how interesting properties of neural processes such as criticality and metastability are extended and amplified when they are embedded in ongoing embodied sensorimotor loops. In such scenarios, minimal models of brain-body-environment dynamical regulation in adaptive behavior, such as the one presented here, offer a conceptual basis for facing complex analysis in real animals due to the low dimensionality of their dynamics. Even models that have little connection with biological brains can provide insights into how neural and sensorimotor dynamics may interact (e.g., extending the range of metastable states, creating asymmetrical loops of causal interaction, etc.), as well as contribute to the development of conceptual methodological tools for understanding the role of the different scales of the system (e.g., the situated vs. passively-coupled comparison to address the role of sensorimotor regulation in the nervous system). Moreover, the availability of real data of brain-bodyenvironment interaction will provide the opportunity to advance

# REFERENCES


in the design of more accurate and realistic models, bringing us closer to capturing fundamental aspects of adaptive behavior.

# AUTHOR CONTRIBUTIONS

Conceived and designed the experiments: MA, MB, XB. Performed the experiments: MA. Analyzed the data: MA. Wrote the paper: MA, MB, XB.

# FUNDING

This research has been partially supported by the project PSI2014-62092-EXP of the National Programme for Fostering Excellence in Scientific and Technical Research ("Explora Ciencia" call) from the Spanish Ministry of Economy and Competitiveness.

# ACKNOWLEDGMENTS

Thanks to Ezequiel Di Paolo, Seth Bullock, and Ricard Solé for useful comments and discussion on some of the results presented in this paper.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Aguilera, Bedia and Barandiaran. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Metastable dynamics in heterogeneous neural fields

#### Cordula Schwappach1, 2, Axel Hutt 3, 4, 5 and Peter beim Graben1, 6 \*

<sup>1</sup> Department of German Studies and Linguistics, Humboldt-Universität zu Berlin, Berlin, Germany, <sup>2</sup> Department of Physics, Humboldt-Universität zu Berlin, Berlin, Germany, <sup>3</sup> Team Neurosys, Inria, Villers-les-Nancy, France, <sup>4</sup> Team Neurosys, Centre National de la Recherche Scientifique, UMR no 7503, Loria, Villers-les-Nancy, France, ¯ <sup>5</sup> Team Neurosys, UMR no 7503, ¯ Loria, Université de Lorraine, Villers-les-Nancy, France, <sup>6</sup> Bernstein Center for Computational Neuroscience, Humboldt-Universität zu Berlin, Berlin, Germany

We present numerical simulations of metastable states in heterogeneous neural fields that are connected along heteroclinic orbits. Such trajectories are possible representations of transient neural activity as observed, for example, in the electroencephalogram. Based on previous theoretical findings on learning algorithms for neural fields, we directly construct synaptic weight kernels from Lotka-Volterra neural population dynamics without supervised training approaches. We deliver a MATLAB neural field toolbox validated by two examples of one- and two-dimensional neural fields. We demonstrate trial-to-trial variability and distributed representations in our simulations which might therefore be regarded as a proof-of-concept for more advanced neural field models of metastable dynamics in neurophysiological data.

#### Edited by:

Emili Balaguer-Ballester, Bournemouth University, UK

#### Reviewed by:

Pablo Varona, Universidad Autonoma de Madrid, Spain Basabdatta Sen Bhattacharya, University of Lincoln, UK

#### \*Correspondence:

Peter beim Graben, Department of German Studies and Linguistics, Humboldt-Universität zu Berlin, Unter den Linden 6, D–10099 Berlin, Germany peter.beim.graben@hu-berlin.de

> Received: 26 March 2015 Accepted: 15 June 2015 Published: 30 June 2015

#### Citation:

Schwappach C, Hutt A and beim Graben P (2015) Metastable dynamics in heterogeneous neural fields. Front. Syst. Neurosci. 9:97. doi: 10.3389/fnsys.2015.00097 Keywords: neural fields, kernel construction, metastability, heteroclinic orbits, trial-to-trial variability, distributed representations, sub-networks, sparsity

# 1. Introduction

Metastable states and transient dynamics between metastable states have received increasing interest in the neuroscientific community in recent time. Beginning with Dietrich Lehmann's original idea to identify "atoms of thought" as metastable topographies, so-called brain microstates, in spontaneous and event-related electroencephalograms (EEG) (Lehmann et al., 1987; Lehmann, 1989; Lehmann et al., 2009), experimentalists found accumulating evidence that metastability is tentatively an important organization principle in neurodynamical systems. Mazor and Laurent (2005), e.g., reported metastable states in the locust odor system (cf. Rabinovich et al., 2001, 2008a), while Hudson et al. (2014) found metastability in the local field potentials of rats recovering from anesthesia. For the analysis of human EEG, several segmentation techniques into metastable states have recently been suggested by Hutt (2004), Allefeld et al. (2009), and beim Graben and Hutt (2015).

From a theoretical perspective, metastable EEG topographies or components of the eventrelated potential (ERP) have been identified with saddle-nodes in deterministic low-dimensional systems by Hutt et al. (2000) and Hutt and Riedel (2003). Particularly, the discoveries of winnerless competition (Rabinovich et al., 2001; Seliger et al., 2003) and heteroclinic orbits in neural population dynamics (Afraimovich et al., 2004a,b; Rabinovich et al., 2008b) led to better understanding of metastability and transient behavior in theoretical neuroscience. Winnerless competition is ubiquitous in complex excitation-inhibition networks with strong asymmetries. While symmetric connectivity usually leads to Hopfield-type attractor neural networks (Hopfield, 1982; Hertz et al., 1991) where transient dynamics is only observed for the motion from a basin of attraction toward an asymptotically stable fixed point attractor, winnerless competition between neural Lotka-Volterra populations (Fukai and Tanaka, 1997; Cowan, 2014) allows for hierarchical transient computations, bifurcations, and the resolution of sequential decision problems, as applied for modeling speech processing (Kiebel et al., 2009), bird songs (Yildiz and Kiebel, 2011), syntactic parsing (beim Graben and Potthast, 2012), and, most recently, working memory (Rabinovich et al., 2014a,b).

However, these phenomena have been investigated on the rather abstract level of macroscopic neural populations so far, without reference to the mesoscopic and microscopic levels of spatially given nervous tissue and individual neurons. One important approach to characterize the former, nervous tissue at the mesoscopic scale, are neural fields, i.e., continuum approximations of infinitely large neural networks (Coombes et al., 2014). In a recent theoretical study, beim Graben and Hutt (2014) investigated stationary states and heteroclinic dynamics in neural fields with heterogeneous synaptic connectivity. The present work applies this previous study to describe experimentally observed transient neural activity as a proofof-concept of our theoretical approach. We propose a novel hypothesis on the origin of trial-to-trial variability observed in most experimental data, on episodic cell assembly dynamics and on sparsely sampled neural representations.

Moreover, we disseminate our software implementation as a MATLAB neural field toolbox to facilitate further research on this intriguing field of computational neuroscience.

### 2. Materials and Methods

In this section we present some of the theoretical findings of beim Graben and Hutt (2014) and indicate how they have been implemented in our simulations.

#### 2.1. Theoretical Background

An important representative of neural fields is given through the Amari equation

$$\frac{\partial u(\mathbf{x},t)}{\partial t} = -u(\mathbf{x},t) + \int\_{\Omega} w(\mathbf{x},\mathbf{y}) f(u(\mathbf{y},t)) \, \mathrm{d}\mathbf{y} \tag{1}$$

describing the evolution of neural activity u(x, t) at site x ∈ ⊂ R d and time t (Amari, 1977). Here, is a d-dimensional manifold, representing neural tissue. Moreover, w(x, y) is the synaptic weight kernel, and f is a sigmoidal activation function, usually chosen as f(u) = 1/(1 + exp(−β(u − θ))), with gain β > 0, and threshold θ > 0. The time scale of the dynamics, often characterized by a particular time constant is implicitly included in the kernel w(x, y).

The neural field described by Equation (1) is called homogeneous when the kernel is translation invariant: w(x, y) = w(x − y). If the field is not homogeneous it is called heterogeneous.

Stationary states, v(x), of the Amari equation which are obtained from ∂u/∂t = 0 obey the nonlinear Hammerstein integral equation

$$\nu(\mathbf{x}) = \int\_{\Omega} \mathbf{w}(\mathbf{x}, \mathbf{y}) f(\nu(\mathbf{y})) \, \mathrm{d}\mathbf{y} \,. \tag{2}$$

By choosing a heterogeneous Pincherle-Goursat kernel (Veltz and Faugeras, 2010)

$$\left(\nu(\mathbf{x},\boldsymbol{\uprho})=\nu(\mathbf{x})\nu(\boldsymbol{\uprho})\right),\tag{3}$$

and carrying out a linear stability analysis, beim Graben and Hutt (2014) were able to prove that the stationary state v(x) is either an asymptotically stable fixed point attractor, or a saddle with a one-dimensional unstable manifold, i.e., a metastable state. Since such saddles could be connected along their stable and unstable directions, heterogeneous neural fields may exhibit stable heteroclinic sequences (SHS: Afraimovich et al., 2004b; Rabinovich et al., 2008b).

Let {v<sup>k</sup> (x)}, 1 ≤ k ≤ n be such a collection of metastable states which we assume to be linearly independent. Then, this collection possesses a biorthogonal system of adjoints {v + k (x)} obeying

$$\int\_{\Omega} \nu\_{\boldsymbol{j}}^{+}(\boldsymbol{\alpha}) \nu\_{\boldsymbol{k}}(\boldsymbol{\alpha}) \, \mathrm{d}\boldsymbol{\alpha} = \delta\_{\boldsymbol{j}\boldsymbol{k}} \, . \tag{4}$$

For the particular case of Lotka-Volterra neural populations, described by activities ξ<sup>k</sup> (t),

$$\frac{\mathrm{d}\xi\_k}{\mathrm{d}t} = \xi\_k \left(\sigma\_k - \sum\_{j=1}^n \rho\_{kj}\xi\_j\right) \tag{5}$$

with growth rates σ<sup>k</sup> > 0, interaction weights ρkj > 0 and ρkk = 1 that are tuned according to the algorithm of Afraimovich et al. (2004b) and Rabinovich et al. (2008b), the population amplitude

$$\alpha\_k(t) = \frac{\xi\_k}{\sigma\_k} \tag{6}$$

recruits its corresponding metastable state v<sup>k</sup> (x), leading to an order parameter expansion

$$\mu(\mathbf{x}, t) = \sum\_{k=1}^{n} \alpha\_k(t) \nu\_k(\mathbf{x}) \tag{7}$$

of the neural field.

Under these assumptions, beim Graben and Potthast (2012) and beim Graben and Hutt (2014) have explicitly constructed the kernel w(x, y) through a power series expansion of the righthand-side of the Amari equation (Equation 1),

$$\frac{\partial \mu(\mathbf{x},t)}{\partial t} = -\mu(\mathbf{x},t) + \int\_{\Omega} \mathbf{w}\_1(\mathbf{x},\mathbf{y}) \mu(\mathbf{y},t) \, \mathrm{d}\mathbf{y} \tag{8}$$

$$+ \int\_{\Omega} \int\_{\Omega} \mathbf{w}\_2(\mathbf{x},\mathbf{y},z) \mu(\mathbf{y},t) \mu(z,t) \, \mathrm{d}\mathbf{y} \, \mathrm{d}z$$

with Pincherle-Goursat kernels<sup>1</sup> .

$$\nu\_1(\mathbf{x}, \boldsymbol{\uprho}) = \sum\_k (\sigma\_k + 1) \nu\_k(\mathbf{x}) \nu\_k^+(\mathbf{y}) \tag{9}$$

$$\left(\boldsymbol{\nu}\_{2}(\boldsymbol{x},\boldsymbol{y},\boldsymbol{z})\right) = -\sum\_{k\boldsymbol{j}} \sigma\_{\boldsymbol{j}} \rho\_{k\boldsymbol{j}} \boldsymbol{\nu}\_{k}(\boldsymbol{x}) \boldsymbol{\nu}\_{k}^{+}(\boldsymbol{y}) \boldsymbol{\nu}\_{\boldsymbol{j}}^{+}(\boldsymbol{z})\,. \tag{10}$$

Interestingly, the kernel w1(x, y) describes a Hebbian synapse between sites y and x whereas the three-point kernel w2(x, y, z) further generalizes Hebbian learning to interactions between three sites x, y, z of neural tissue.

#### 2.2. Numerical Studies

For a numerical implementation of the theoretical results above, we have to discretize time and space. Using MATLAB, temporal discretization on the one hand is achieved through the ordinary differential equation solver ode15s for stiff problems. On the other hand, spatial discretization converts the kernels w<sup>1</sup> and w<sup>2</sup> into tensors of rank two and three, respectively. Consequently, the integrals in Equation (8) become contractions over products of tensors and state vectors u(t). In order to properly deal with tensor algebra, we use the Sandia Tensor Toolbox<sup>2</sup> . Our neural field toolbox, thus obtained is available as Supplementary Material. We evaluate our implementation in the next subsections by means of two examples.

#### 2.2.1. One-dimensional Neural Field

In our first simulation, we use a d = 1 dimensional neural field where we choose n = 3 sine functions

$$\nu\_k(\mathbf{x}) = \sin k\mathbf{x} \tag{11}$$

as metastable states on the domain = [0, 2π] discretized with a spatial grid of N<sup>x</sup> = 100 sites. According to the orthogonality relations

$$\int\_{\Omega} \sin jx \,\sin kx \,\mathrm{d}x = \pi \,\delta\_{jk} \tag{12}$$

we easily obtain the adjoint modes

$$\nu\_k^+(\mathbf{x}) = \frac{1}{\pi} \sin k\mathbf{x}\,. \tag{13}$$

For the temporal dynamics we prepare the stable heteroclinic contour solving (Equation 5) used by beim Graben and Hutt (2015) with σ<sup>1</sup> = 1, σ<sup>2</sup> = 2, σ<sup>3</sup> = 3. Metastable states v<sup>k</sup> (x) and their population activities ξ<sup>k</sup> (t) are shown in **Figure 1**.

We run simulations with one fixed initial condition and also from an ensemble of 60 initial conditions randomly distributed in the vicinity of the first saddle, where we add some small portion of Gaussian observational noise (noise level σ = 0.005) in order to demonstrate trial-to-trial variability and hence event-related phase decoherence (Jung et al., 2001; Makeig et al., 2002).

#### 2.2.2. Two-dimensional Neural Field

For our second demonstration, we assume a spatially distributed response in a neural population to external stimuli triggering a sequence of neural activity patterns. It is well-established that sensory input features (Pasupathy and Connor, 2002) at earlier stages of the object's representation pathway and memory (Rissman and Wagner, 2012) is encoded by distributed cortical neural populations while objects are sparsely coded in later stages of the representation pathway (Connor, 2005). Here we consider a cortical neural population embedded in twodimensional space involving interleaved patterns. These patterns are d = 2 dimensional gray scale bitmap images of the numbers<sup>3</sup> 1, 2, and 3 (see **Figure 4** in Section 3.2). In the implementation, these bitmaps are downsampled to a 20 × 20 grid and reshaped into vectors with N<sup>x</sup> = 400 elements. Adjoint patterns are obtained as Moore-Penrose pseudoinverses (Hertz et al., 1991).

The temporal evolution of these patterns follows the same heteroclinic contour as above. Here, the underlying working assumption is the presence of interacting sub-networks, e.g., reflecting several distributed representations of signal features or of pieces of working memory. The study predicts what one expects to measure in single spatial locations while the neural system encodes information in a spatially distributed population.

### 3. Results

The results of our simulation studies are presented in this section.

#### 3.1. One-dimensional Neural Field

For the one-dimensional neural field we compare in **Figure 2** the prescribed spatiotemporal dynamics as resulting from the order parameter expansion (Equation 7) with the solution of the Amari (Equation 8).

**Figure 2A** shows the prescribed dynamics on a spatiotemporal grid with time on the x-axis and space on the y-axis. The instantaneous activations are therefore given by vertical slices. Going from left to right, these slices first exhibit one wave crest (in red at the bottom) and one wave trough (in blue at the top), corresponding to metastable state v1(x). Around time t = 15 the frequency doubles and metastable state v2(x) can be observed for approximately seven ticks. The third metastable state met by the trajectory around time t = 21 is the mode with tripled frequency. It is only stable for five ticks and evolves thereafter into the first mode again.

In contrast, **Figure 2B** depicts the numerical solution of the Amari equation (Equation 8). Obviously, no deviation is visible.

In order to draw neurophysiologically relevant conclusions from our toy model, we consider the metastable states of the heteroclinic contour as "synthetic ERP components" (Barrès et al., 2013) measured with "electrodes" at the particular sampling points. Because ERPs are obtained from averaging spontaneous EEG over ensembles of several trials that are time-locked to the perception or processing of stimuli, we simulate 60 synthetic ERP trials by randomly preparing initial conditions of the Amari equation.

<sup>1</sup> There was a mistake in our previous reports (beim Graben and Potthast, 2012; beim Graben and Hutt, 2014). Although the kernel construction has been correctly derived, a minus sign was omitted in the final result for kernel w2(x, y, z). This is corrected now.

<sup>2</sup> http://www.sandia.gov/~tgkolda/TensorToolbox/index-2.5.html

<sup>3</sup>Original images are taken from the webpage http://www.iconarchive.com/tag/ number-3 before modifications with respect to color and resolution.

FIGURE 1 | Prescribed dynamics. (A) Three sinusoids as spatial patterns. (B) Stable heteroclinic contour resulting from winnerless competition in a Lotka-Volterra system (Equation 5). Blue: k = 1, green: k = 2, red: k = 3.

The results are displayed in **Figure 3** for four "measurement electrodes" at positions 3, 21, 47, and 88. Interestingly, our algorithm exhibited numerical instabilities in five runs which have been marked as "rejected" outliers and excluded from presentation. The resulting 55 trials are shown as colored traces in **Figure 3**. At simulation start all signals are nicely coherent, but later substantial phase dispersions take place (Jung et al., 2001; Makeig et al., 2002).

We also calculated the ERP averages from our simulation shown as bold black traces in **Figure 3**. On the one hand, the averaged ERP is much smoother than the noisy single realizations which justifies averaging in our simulation. However, the averaged ERP significantly decays in the course of time. This is obviously due to the increasing phase decoherence (Jung et al., 2001; Makeig et al., 2002).

#### 3.2. Two-dimensional Neural Field

The numerical simulation of Equation (8) yields a sequence of two-dimensional transient patterns which is shown as a sampled sequence of snapshot maps in **Figure 4**.

According to the different growth rates σ<sup>k</sup> of the populations, pattern "1" stays the longest period of time, pattern "2" is visible for a shorter period of time and pattern "3" can be seen for the shortest period of time. These modes represent interweaved spatial networks reflecting intrinsically stored activity patterns.

Now assuming that measurement of neural activity takes place at discrete spatial locations (color-coded points in **Figure 4**), one observes different transient dynamics dependent on the spatial location of the measurement point that is shown in **Figure 5**. Considering the red-coded spatial location, one observes strong activity in the time periods when pattern "1" is active, and wellreduced activity in the time windows of active patterns "2" and "3." Conversely, the activity at the blue-coded location defined in **Figure 4** raises only if pattern "3" is active, otherwise its activity is well-reduced. The green-coded spatial location shows negligible activity in time periods when pattern "1" is active while activity is increased during the emergence of patterns "2" and "3."

# 4. Discussion

In this paper, we presented a software implementation (neural field toolbox) and numerical simulation results of previously reported theoretical findings on metastable states and heteroclinic dynamics in neural fields (beim Graben and Potthast, 2012; beim Graben and Hutt, 2014). For the particular case of Lotka-Volterra population dynamics and linearly independent spatial modes, the synaptic weight kernel of the Amari neural field equation (Amari, 1977) can be explicitly constructed from the prescribed metastable states and their evolution parameters as Pincherle-Goursat kernels. This is an important finding as our kernel construction method is not a standard training algorithm such as backpropagation (Igel et al., 2001; beim Graben and Potthast, 2009). Yet it implements a straightforward generalization of Hebbian learning algorithms (beim Graben and Potthast, 2009; Potthast and beim Graben, 2009).

We validated our algorithm by means of two examples, a one-dimensional neural field where metastable states are three sinusoidal excitations over a line, and a two-dimensional example where we have chosen three bitmap images as spatial modes. The temporal dynamics was prescribed as a heteroclinic contour connecting these three patterns in a closed loop. In both simulations, the results were in exact agreement with the prescribed trajectories.

Furthermore, we examined the issues of trial-to-trial variability and distributed representations. In the first example we created solutions for randomly prepared initial conditions, thereby emulating phase resetting in event-related brain potentials (ERP). We observed increasing phase decoherence in the resulting ERP averages. Our model presents a theoretically satisfying explanation for this ubiquitous experimental finding (Jung et al., 2001; Makeig et al., 2002). Assuming that ERP components are metastable states that are connected along heteroclinic orbits (Hutt and Riedel, 2003; beim Graben and Hutt, 2015), single ERP trials start from randomly distributed initial conditions, sometimes closer and sometimes farther from the respective metastable stable. These initial distances from a metastable state result in acceleration and hence in velocity differences in phase space, eventually leading to dispersion and decoherence. Moreover, such a dependence on initial conditions resembles previous experimental results by Pastalkova et al. (2008) showing that identical experimental initial conditions

in a motor task lead to identical sequences of cell assembly activations, while different initial conditions yield different sequences.

For the second example we considered the interaction of three two-dimensional populations, cf. **Figure 4**. The transient passage of the system at metastable attractors has been shown experimentally in previous studies, such as in middle-latent auditory evoked potentials (Hutt and Riedel, 2003) or in the population response of olfactory projection neurons to odor stimuli (Mazor and Laurent, 2005). For instance, the study of Mazor and Laurent (2005) also shows nicely the responses of single neurons in the population revealing different activity in different neurons: some neurons respond to the external stimulus, others remain silent. Such a distinction in response can easily be explained by an insufficient spatial sub-sampling in the measurement and the presence of spatially distributed patterns. However, just spatial subsampling does not explain the fully distinct activity of different neurons, such as different episode neurons found in the hippocampus (Pastalkova et al., 2008). Here, different neurons show distinct episodic temporal activities. The equivalent temporal evolution is shown in our simulations in **Figure 5**, where the units at different spatial locations exhibit different temporal sequences of activation that are highly correlated to the presence of the respective pattern representations. This difference results from interacting populations or cell assemblies.

The latter line of argumentation raises the question whether it may explain previous results on sparse neural representations or even may contribute to the question on the existence of "grandmother cells" (Connor, 2005; Quiroga et al., 2005). At a first glance, the present work assumes the existence of interacting spatially distributed sub-networks and supports their existence by a qualitative comparison to previous experimental results by Mazor and Laurent (2005) and Pastalkova et al. (2008). Our assumption of interacting sub-networks does not rule out sparse neural representations since our modeling approach does not stipulate contiguous spatial patterns but also allows for sparse patterns as well.

Metastable neural field dynamics as an ubiquitous organization principle of the brain is also consistent with findings from neuroanatomy and cognitive neuroscience. Anatomically, neural circuits comprise convergent and divergent pathways between populations (Kandel et al., 1991). Assuming that a particular sub-network gets activated by percolation along a convergent pathway and deactivated along a divergent pathway subsequently entails a saddle-node picture in its phase space description, hence a metastable attractor. In cognitive neuroscience, mental representations are regarded as intermediate results of cognitive computations in discrete time. In order to embed these into continuous physical time, they have to be considered connected through continuous trajectories along their stable and unstable directions, i.e., as metastable states, again (beim Graben and Potthast, 2009, 2012).

The present study is a first step toward metastability in neural fields. We hope that our work encourages further research on metastability in neural fields to describe transient neural dynamics by interacting populations and contribute to the description of neural information storage, being either distributed or sparse.

# Author Contributions

This study reports results from CS's student internship at Department of German Studies and Linguistics, Humboldt-Universität zu Berlin. CS developed the program code and conducted the numerical simulations. AH included the subnetwork study, PbG contributed the study on trial-to-trial variability and compiled the neural field toolbox. All authors wrote the manuscript together.

## Funding

PbG acknowledges support by a Heisenberg Fellowship of the German Research Foundation DFG (GR 3711/1-2) and of the Bernstein Center for Computational Neuroscience, Berlin, hosting AH as visiting professor during October 2014. AH acknowledges funding from the European Research Council for support under the European Union's Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no. 257253.

# Supplementary Material

The Supplementary Material for this article can be found online at: http://journal.frontiersin.org/article/10.3389/fnsys. 2015.00097

#### Supplemental Data

A GIF animation of the two-dimensional neural field simulation is given as Supplementary Material. Moreover, we deliver a MATLAB software neural field toolbox. This package essentially comprises three routines: amarikernels, amarieq, iniamari, and a main program, solveamari to be evoked in the the following way: amarikernels is the training program for the synaptic weights. It takes four arguments: V\_patterns,sigmarange,compbias, and contourflag and returns two kernel tensors K1, K2. V\_patterns is a matrix whose columns are the metastable states in a spatial discretization, their order corresponds to the desired heteroclinic sequence. sigmarange is an interval of Lotka-Volterra grow rates σ characterizing the time scale of the dynamics, while compbias denotes the competition bias in the interaction matrix (ρ). The last parameter, contourflag, is a binary flag deciding whether the hereroclinic sequence is closed (1) or not (0). A closed hereroclinic sequence is called hereroclinic contour. amarieq defines the Amari equation (Equation 8) for the ODE solver. It has four input arguments, t,V, K1, K2, where t is the time span to be simulated, V is the actual field activity, and K1, K2 are the two synaptic weight kernels. iniamari prepares an initial condition at the surface of the simplex spanned by the metastable states V\_patterns. The other arguments are lead and remain, denoting the leading direction toward the next saddle and its orthogonal projection on the remaining modes. Finally, solveamari presents the code for our onedimensional neural field example, evoking the ODE solver in line 48.

# References


Amari, S.-I. (1977). Dynamics of pattern formation in lateral-inhibition type neural fields. Biol. Cybern. 27, 77–87.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Schwappach, Hutt and beim Graben. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Stimuli Reduce the Dimensionality of Cortical Activity

#### Luca Mazzucato<sup>1</sup> , Alfredo Fontanini 1, 2 and Giancarlo La Camera1, 2 \*

<sup>1</sup> Department of Neurobiology and Behavior, State University of New York at Stony Brook, Stony Brook, NY, USA, <sup>2</sup> Graduate Program in Neuroscience, State University of New York at Stony Brook, Stony Brook, NY, USA

The activity of ensembles of simultaneously recorded neurons can be represented as a set of points in the space of firing rates. Even though the dimension of this space is equal to the ensemble size, neural activity can be effectively localized on smaller subspaces. The dimensionality of the neural space is an important determinant of the computational tasks supported by the neural activity. Here, we investigate the dimensionality of neural ensembles from the sensory cortex of alert rats during periods of ongoing (inter-trial) and stimulus-evoked activity. We find that dimensionality grows linearly with ensemble size, and grows significantly faster during ongoing activity compared to evoked activity. We explain these results using a spiking network model based on a clustered architecture. The model captures the difference in growth rate between ongoing and evoked activity and predicts a characteristic scaling with ensemble size that could be tested in high-density multi-electrode recordings. Moreover, we present a simple theory that predicts the existence of an upper bound on dimensionality. This upper bound is inversely proportional to the amount of pair-wise correlations and, compared to a homogeneous network without clusters, it is larger by a factor equal to the number of clusters. The empirical estimation of such bounds depends on the number and duration of trials and is well predicted by the theory. Together, these results provide a framework to analyze neural dimensionality in alert animals, its behavior under stimulus presentation, and its theoretical dependence on ensemble size, number of clusters, and correlations in spiking network models.

Edited by:

Ruben Moreno-Bote, Universitat Pompeu Fabra, Spain

#### Reviewed by:

Martin Paul Nawrot, Universität zu Köln, Germany Iñigo Arandia-Romero, Universitat Pompeu Fabra, Spain

#### \*Correspondence:

Giancarlo La Camera giancarlo.lacamera@stonybrook.edu

> Received: 31 August 2015 Accepted: 02 February 2016 Published: 17 February 2016

#### Citation:

Mazzucato L, Fontanini A and La Camera G (2016) Stimuli Reduce the Dimensionality of Cortical Activity. Front. Syst. Neurosci. 10:11. doi: 10.3389/fnsys.2016.00011 Keywords: gustatory cortex, dimensionality, hidden markov models, ongoing activity, mean field theory, spiking network model, metastable dynamics

# INTRODUCTION

Understanding the dynamics of neural activity and how it is generated in cortical circuits is a fundamental question in Neuroscience. The spiking activity of ensembles of simultaneously recorded neurons can be represented in terms of sequences of firing rate vectors, as shown e.g., in frontal (Abeles et al., 1995; Seidemann et al., 1996; Durstewitz et al., 2010), gustatory (Jones et al., 2007; Mazzucato et al., 2015), motor (Kemere et al., 2008), premotor and somatosensory cortex (Ponce-Alvarez et al., 2012). The dimension of each firing rate vector is equal to the number of ensemble neurons N and the collection of rate vectors across trials takes the form of a set of points in the N-dimensional space of firing rates. Such points may not fill the whole space, but be restricted to lie inside a lower-dimensional subspace (see Ganguli et al., 2008). Roughly, dimensionality is the minimal number of dimensions necessary to provide an accurate description of the neural dynamics. If ensemble neurons are independent of each other, neural activities at different times will scatter around in the space of firing rate, filling a large portion of the space. In this case, dimensionality will be maximal and equal to the size of the ensemble N. At the other extreme, if all neurons are strongly correlated, ensemble activity localizes along a line. In this case, dimensionality is minimal and equal to one. These simple examples suggest that dimensionality captures information about the structure of a cortical circuit and the functional relations among the simultaneously recorded neurons, such as their firing rates correlation computed over timescales of hundreds of milliseconds.

Different definitions of dimensionality have been introduced for different tasks and across neural systems (Ganguli et al., 2008; Churchland et al., 2010a; Abbott et al., 2011; Ganguli and Sompolinsky, 2012; Cadieu et al., 2013; Rigotti et al., 2013; Gao and Ganguli, 2015). Such measures of dimensionality can shed light on the underlying neural computation; for example, they can predict the onset of an error trial in a recall task (Rigotti et al., 2013), or can allow the comparison of classification accuracy between different brain areas (e.g., IT vs. V4) and synthetic algorithms (Cadieu et al., 2013). Here, we investigate a measure of dimensionality closely related to the firing rate correlations of simultaneously recorded neurons (Abbott et al., 2011); such correlations may provide a signature of feature-based attention (Cohen and Maunsell, 2009) and other top-down cognitive factors (Nienborg et al., 2012). We elucidate the dependence of dimensionality on experimental parameters, such as ensemble size and interval length, and we show that it varies across experimental conditions. We address these issues by comparing recordings of ensembles of neurons from the gustatory cortex (GC) of alerts rats to a biologically plausible network model based on neural clusters with recurrent connectivity. This model captures neural activity in GC during periods of ongoing and stimulus-evoked activity, explaining how the spatiotemporal dynamics of ensemble activity is organized in sequences of metastable states and how single-neuron firing rate distributions are modulated by stimulus presentation (Mazzucato et al., 2015). Here, we show that the same model expounds the observed dependence of dimensionality on ensemble size and how such dependence is reduced by the presentation of a stimulus. By comparing the clustered network model with a homogeneous network without clusters, we find that the clustered network has a larger dimensionality that depends on the number of clusters and the firing rate correlations among ensemble neurons. A simple theory explains these results and allows extrapolating the scaling of dimensionality to very large ensembles. Our theory shows that recurrent networks with clustered connectivity provide a substrate for high-dimensional neural representations, which may lead to computational advantages.

# METHODS

# Experimental Procedures

Adult female Long Evans rats were used for this study (Samuelsen et al., 2012; Mazzucato et al., 2015). Animals received ad lib. access to food and water, unless otherwise mentioned. Movable bundles of 16 microwires attached to a "mini-microdrive" (Fontanini and Katz, 2006; Samuelsen et al., 2012) were implanted in GC (AP 1.4, ML ± 5 from bregma, DV –4.5 from dura). After electrode implantation, intra-oral cannulae (IOC) were inserted bilaterally (Phillips and Norgren, 1970; Fontanini and Katz, 2005). At the end of the surgery a positioning bolt for restraint was cemented in the acrylic cap. Rats were given at least 7 days for recovery before starting the behavioral procedures outlined below. All experimental procedures were approved by the Institutional Animal Care and Use Committee of Stony Brook University and complied with University, state, and federal regulations on the care and use of laboratory animals. More details can be found in Samuelsen et al. (2012).

Rats were habituated to being restrained and receiving fluids through IOCs, and then trained to self-deliver water by pressing a lever following a 75 dB auditory cue at a frequency of 4 KHz. The interval at which lever-pressing delivered water was progressively increased to 40 ± 3 s (ITI). During experimental sessions additional tastants were automatically delivered at random times near the middle of the ITI, at random trials and in the absence of the anticipatory cue. A computer-controlled, pressurized, solenoid-based system delivered ∼40µl of fluids (opening time ∼40 ms) directly into the mouth through a manifold of 4 polymide tubes slid into the IOC. The following four tastants were delivered: 100 mM NaCl, 100 mM sucrose, 100 mM citric acid, and 1 mM quinine HCl. Water (∼50µl) was delivered to rinse the mouth clean through a second IOC 5 s after the delivery of each tastant. Each tastant was delivered for at least 6 trials in each condition. Upon termination of each recording session the electrodes were lowered by at least 150µm so that a new ensemble could be recorded.

Evoked activity periods were defined as the interval after tastant delivery (time t = 0 in our figures) and before water rinse (time t = 5 s). Only trials in which the tastants were automatically delivered were considered for the analysis of evoked activity, to minimize the effects of cue-related expectations (Samuelsen et al., 2012). Ongoing activity periods were defined as the 5 s-long intervals at the end of each inter-trial period.

The behavioral state of the rat was monitored during the experiment for signs of disengagement. Erratic lever pressing, inconstant mouth movements and fluids dripping from the mouth indicated disengagement and led to the termination of the experiment. In addition, since disengagement from the task is also reflected in the emergence of high power µ oscillations in local field potentials, occurrences of such periods were removed offline and not analyzed further (Fontanini and Katz, 2008).

# Data Analysis

Single neuron action potentials were amplified, bandpass filtered (at 300–8 KHz), digitized and recorded to a computer (Plexon, Dallas, TX). Single units of at least 3:1 signal-to-noise ratio were isolated using a template-matching algorithm, cluster cutting techniques and examination of inter-spike interval plots (Offline Sorter, Plexon, Dallas, TX). All data analyses and model simulations were performed using custom software written in Matlab (Mathworks, Natick, MA, USA), Mathematica (Wolfram Research, Champaign, IL), and C. Starting from a pool of 299 single neurons in 37 sessions, neurons with peak firing rate lower than 1 Hz (defined as silent) were excluded from further analysis, as well as neurons with a large peak around the 6–10 Hz in the spike power spectrum, which were considered somatosensory (Katz et al., 2001; Samuelsen et al., 2012; Horst and Laubach, 2013). Only ensembles with 3 or more simultaneously recorded neurons were further analyzed (167 non-silent, non-somatosensory neurons from 27 ensembles). We analyzed ongoing activity in the 5 s interval preceding either the auditory cue or taste delivery, and evoked activity in the 5 s interval following taste delivery in trials without anticipatory cue, wherein significant taste-related information is present (Jezzini et al., 2013).

### Hidden Markov Model (HMM) Analysis

Here we briefly outline the procedure used in Mazzucato et al. (2015), see this reference and (Jones et al., 2007; Escola et al., 2011; Ponce-Alvarez et al., 2012) for further details. Under the HMM, a system of N recorded neurons is assumed to be in one of a predetermined number of hidden (or latent) states (Rabiner, 1989; Zucchini and MacDonald, 2009). Each state m is defined as a vector of N firing rates ν<sup>i</sup> (m), i = 1, . . . , N, one for each simultaneously recorded neuron. In each state, the neurons were assumed to discharge as stationary Poisson processes (Poisson-HMM). We matched the model to the data segmented in 1-ms bins (see below). In such short bins, we found that typically at most one spike was emitted across all simultaneously recorded neurons. If more than one neuron fired an action potential in a given bin, only one (randomly chosen) was kept for further analysis (this only occurred in a handful of bins per trial; Escola et al., 2011). We denote by y<sup>i</sup> (t) the spiking activity of the i-th neuron in the interval [t, t + dt], y<sup>i</sup> (t) = 1 if the neuron emitted a spike and y<sup>i</sup> (t) = 0 otherwise. Denoting with S<sup>t</sup> the hidden state of the ensemble at time t, the probability of having a spikes from neuron i in a given state m in the interval [t, t + dt] is given by p y<sup>i</sup> (t) = 1 <sup>S</sup><sup>t</sup> <sup>=</sup> <sup>m</sup> = 1 − e νi(m)dt .

The firing rates ν<sup>i</sup> (m) completely define the states and are also called "emission probabilities" in HMM parlance. The emission and transition probabilities were found by maximization of the log-likelihood of the data given the model via the expectationmaximization (EM), or Baum-Welch, algorithm (Rabiner, 1989), a procedure known as "training the HMM." For each session and type of activity (ongoing vs. evoked), ensemble spiking activity from all trials was binned at 1 ms intervals prior to training assuming a fixed number of hidden states M (Jones et al., 2007; Escola et al., 2011). For each given number of states M, the Baum-Welch algorithm was run 5 times, each time with random initial conditions for the transition and emission probabilities. The range of hidden states M for the HMM analyses were Mmin = 10 and Mmax = 20 for spontaneous activity, and Mmin = 10 and Mmax = 40 for evoked activity. Such numbers were based on extensive exploration of the parameter space and previous studies (Jones et al., 2007; Miller and Katz, 2010; Escola et al., 2011; Ponce-Alvarez et al., 2012; Mazzucato et al., 2015). For evoked activity, each HMM was trained on all four tastes simultaneously. Of the models thus obtained, the one with largest total likelihood M<sup>∗</sup> was taken as the best HMM match to the data, and then used to estimate the probability of the states given the model and the observations in each bin of each trial (a procedure known as "decoding"). During decoding, only those hidden states with probability exceeding 80% in at least 50 consecutive bins were retained (henceforth denoted simply as "states"). State durations were approximately exponentially distributed with median duration 0.60 s (95% CIs: 0.07–4.70) during ongoing activity and 0.30 s (0.06–2.80) during evoked activity (Mazzucato et al., 2015).

The firing rate fits ν<sup>i</sup> (m) in each trial were obtained from the analytical solution of the maximization step of the Baum-Welch algorithm,

$$\psi\_i(m) = -\frac{1}{dt} \ln \left( 1 - \frac{\sum\_{t=1}^T r\_m(t) \, \boldsymbol{y}\_i(t)}{\sum\_{t=1}^T r\_m(t)} \right). \tag{1}$$

Here, [yi(1), . . . , yi(T)] is the spike train of the i-th neuron in the current trial, and T is the total duration of the trial. r<sup>m</sup> (t) = P(S<sup>t</sup> = m|y(1), . . . , y(T)) is the probability that the hidden state S<sup>t</sup> at time t is m, given the observations.

## Dimensionality Measure

We defined the dimensionality of the neural activity as

$$d = \frac{1}{\sum\_{i=1}^{N} \widetilde{\lambda}\_i^2},\tag{2}$$

where the <sup>e</sup>λ<sup>i</sup> are the principal eigenvalues expressed as fractions of the total amount of variance explained, i.e., <sup>e</sup>λ<sup>i</sup> <sup>=</sup> <sup>λ</sup>i/ ( P <sup>j</sup> λj), where λ<sup>j</sup> are the eigenvalues of the covariance matrix of the firing rates (see below).

The dimensionality can be computed exactly in some relevant special cases. The calculation is simplified by the observation that Equation (2) is equivalent to

$$d = \frac{\left[Tr\left(C\_f\right)\right]^2}{Tr(C\_f^2)},$$

where C<sup>f</sup> is the true covariance matrix of the firing rate vectors, Tr (A) ≡ P<sup>N</sup> <sup>i</sup>=<sup>1</sup> Aii is the trace of matrix A, and Tr A 2 = P<sup>N</sup> <sup>i</sup>,j=<sup>1</sup> AijAji. We consider in the following only the case of firing rates in equal bins, hence we can replace C<sup>f</sup> with the covariance matrix of the spike counts C in the definition of d:

$$d = \frac{[Tr(\mathcal{C})]^2}{Tr(\mathcal{C}^2)} = \frac{b\_N^2}{c\_N + a\_N},\tag{3}$$

where for later convenience we have introduced the notation

$$a\_N = \sum\_{i=1}^N \mathcal{C}\_{ii}^2, \; b\_N = \sum\_{i=1}^N \mathcal{C}\_{ii}, \; c\_N = \sum\_{i \neq j}^N \mathcal{C}\_{ij} \mathcal{C}\_{ji}. \tag{4}$$

Note that d does not depend on the distribution of firing rates, but only on their covariance, up to a common scaling factor.

#### Dimensionality in the case of uniform pair-wise correlations

When all the pair-wise correlations rijare identical, rij = ρ for all i 6= j,

$$r\_{ij} = \begin{pmatrix} 1 & \rho & \dots \dots \rho \\ \rho & 1 & & \vdots \\ \vdots & & \ddots & \rho \\ \rho & \dots & \rho & 1 \end{pmatrix} \tag{5}$$

we have Cij = ρ q σ 2 i σ 2 j for i 6= j, where σ 2 <sup>i</sup> = Cii is the spike count variance. In this case, we find from Equation (4) that

$$a\_N = \sum\_{i=1}^N \sigma\_i^4, \ b\_N = \sum\_{i=1}^N \sigma\_i^2, \ c\_N = \rho^2 (b\_N^2 - a\_N). \tag{6}$$

and the dimensionality, Equation (3), is given by

$$d = \frac{1}{\rho^2 + \left(1 - \rho^2\right)g(N)},\tag{7}$$

where

$$\mathbf{g}\_N = \frac{a\_N}{b\_N^2} = \frac{\sum\_{i=1}^N \sigma\_i^4}{\left(\sum\_{i=1}^N \sigma\_i^2\right)^2},$$

Note that since both a<sup>N</sup> and b<sup>N</sup> scale as N when N is large, in general g<sup>N</sup> ∼ 1 N for large N.

If all spike counts have equal variance, σ<sup>i</sup> = σ, we find exactly g<sup>N</sup> = 1 N :

$$d = \frac{1}{\rho^2 + \frac{\{1 - \rho^2\}}{N}} = \frac{N}{N\rho^2 + \{1 - \rho^2\}},\tag{8}$$

and the dependence of d on the variance drops out. Note that for uncorrelated spike counts (ρ = 0) this formula gives d = N, whereas for any finite correlation we find the upper bound d = 1/ρ<sup>2</sup> . For N > 1, the dimensionality is inversely related to the amount of pair-wise correlation ρ.

Consider the case where spike counts have variances σ 2 i drawn from a probability distribution with mean E - σ 2 i = σ 2 and variance Var- σ 2 i = δσ 4 , and the pair-wise correlation coefficients rij, for i 6= j, are drawn from a distribution with mean E - rij = ρ and variance Var- rij = δρ 2 . In such a case one can evaluate Equation (3) approximately by its Taylor expansion around the mean values of the quantities in Equation (4). At leading order in N one finds

$$E\left[d\right] \approx \frac{E\left[b\_N^2\right]}{E\left[c\_N\right] + E\left[a\_N\right]} = \frac{N\sigma^4 + 8\sigma^4}{\left(N-1\right)\sigma^4\left(\rho^2 + 8\rho^2\right) + \sigma^4 + 8\sigma^4},\tag{9}$$

where E[.] denotes expectation. To obtain this result we have used the definitions in Equation (4), from which

$$E[\mathbf{a}\_N] = N(\sigma^4 + 8\sigma^4), \ E[b\_N^2] = N^2 \sigma^4 + N8\sigma^4,$$

$$E[\mathbf{c}\_N] = (N^2 - N)\sigma^4(\rho^2 + 8\rho^2), \tag{10}$$

and the fact that, given a random vector X<sup>i</sup> with mean µ<sup>i</sup> and covariance Cij, and a constant symmetric matrix Aij, the expectation value of the quadratic form P <sup>i</sup>,<sup>j</sup> XiAijX<sup>j</sup> is

$$E[\sum\_{i,j} X\_i A\_{ij} X\_j] = \sum\_{ij} (A\_{ij} \mathbf{C}\_{ji} + \mu\_i A\_{ij} \mu\_j). \tag{11}$$

In the case of uncorrelated spike counts (ρ = 0, δρ = 0), dimensionality still depends linearly on the ensemble size N, but with a smaller slope <sup>σ</sup> 4 σ 4+δσ <sup>4</sup> < 1 compared to the case of equal variances (Equation 8 with ρ = 0).

#### Dimensionality in the Case of Neural Clusters

Given an ensemble of N neurons arranged in Q clusters (motivated by the model network described later in section "Spiking neuron model"), we created ensembles of uncorrelated spike trains for N ≤ Q and correlated within each cluster for N > Q. Thus, if N ≤ Q the correlation matrix is the N × N identity matrix. If N > Q, the (Q+1)th neuron was added to the first cluster, with correlation ρ with the other neuron of the cluster, and uncorrelated to the neurons in the remaining clusters. The (Q+2)th neuron was added to the second cluster, with correlation ρ with the other neuron of the second cluster, and uncorrelated to the neurons in the remaining clusters, and so on. Similarly, the (2Q+p)th neuron (p ≤ Q) was added to the p-th cluster, with pair-wise correlation ρ with the other neurons of the same cluster, but no correlation with the neurons in the remaining clusters; and so on. In general, for N = mQ + p neurons (where m = h N Q i − ≥ 1 is the largest integer smaller than <sup>N</sup> Q ), the procedure picked m + 1 neurons per cluster for the first p cluster and m neurons per cluster for the remaining Q − p clusters, with uniform pair-wise correlations ρ in the same cluster while neurons from different clusters were uncorrelated. The resulting correlation matrix r was block diagonal

$$r = \operatorname{diag} \left( R\_1, \dots, R\_Q \right),$$

where each of the Q blocks contains the correlations of neurons from the same cluster. Inside each block R<sup>i</sup> , the off-diagonal terms are equal to the uniform within-cluster correlation ρ:

$$R\_{\vec{i}} = \begin{pmatrix} 1 & \rho & \dots & \rho \\ \rho & 1 & & \vdots \\ \vdots & & \ddots & \rho \\ \rho & \dots & \rho & 1 \end{pmatrix}$$

The first p blocks have size (m + 1) × (m + 1) and the last Q − p blocks have size m × m, so that (m + 1) p + m Q − p = N. The remaining elements of matrix r (representing pair-wise correlations of neurons belonging to different clusters) were all zero. Recalling that Cij = rijσiσ<sup>j</sup> , one finds Tr (C) = pbm+<sup>1</sup> + Q − p b<sup>m</sup> andTr C 2 = ρ 2 [pb<sup>2</sup> <sup>m</sup>+<sup>1</sup> + Q − p b 2 <sup>m</sup>] + (1 − ρ 2 )[pam+<sup>1</sup> + Q − p am], where a<sup>n</sup> and b<sup>n</sup> are defined in Equation (6), from which one obtains

$$d = \begin{cases} \frac{b\_N^2 / a\_N}{\left[\rho b\_{m+1} + (Q-\rho)b\_m\right]^2} & , \ N \le Q\\ \frac{\left[\rho b\_{m+1} + (Q-\rho)b\_m\right]^2}{\rho^2 \left[\rho b\_{m+1}^2 + (Q-\rho)b\_m^2\right] + (1-\rho^2)\left[\rho a\_{m+1} + (Q-\rho)a\_m\right]} & , \ N > Q \end{cases} \tag{12}$$

In the approximation where all neurons have the same variance this simplifies to

$$d = \begin{cases} N & \text{, } N \le Q \\ \frac{N}{1 + m\rho^2 \left[1 - (Q - \rho)/N\right]}, & N > Q. \end{cases} \tag{13}$$

Recall that in the formulae above m and p depend on N. For finite ρ, Equation (13) predicts the bound d ≤ Q/ρ<sup>2</sup> for any N > 1, with this value reached asymptotically for large N. When single neuron variances σ 2 i are drawn from a distribution with mean E - σ 2 i = σ 2 and variance Var- σ 2 i = δσ 4 , an expression for the dimensionality can be obtained from Equation (12) at leading order in the expectation values of the quantities in Equation (4) (not shown), with a procedure similar to that used to obtain Equation (9).

# Pair-Wise Correlations

Given neuron i and neuron j's spike trains, we computed the spike count correlation coefficient rij

$$r\_{\vec{\eta}} = \frac{\mathcal{S}\_{\vec{\eta}}}{\sqrt{\mathcal{S}\_{\vec{\imath}\vec{\imath}}\mathcal{S}\_{\vec{\jmath}\vec{\jmath}}}},$$

where S is the sample covariance matrix of the spike counts estimated as

$$S\_{ij} = \frac{1}{N\_b N\_T - 1} \sum\_{b,s=1}^{N\_b, N\_T} \left( n\_i \left( b, s \right) - < n\_i > \right) \left( n\_j \left( b, s \right) - < n\_j > \right),\tag{14}$$

where ni(b,s) is the spike count of neuron i in bin b and trial s. The sum goes over all N<sup>b</sup> bins and over all N<sup>T</sup> trials in a session, whereas < n<sup>i</sup> > is the average across trials and bins for neuron i. In the main text and figures we present results obtained with a bin size of 200 ms, but have performed the same analyses with bin sizes varying from 10 ms to 5 s (see Results for details).

Significance of the correlation was estimated as follows (Renart et al., 2010): Nshuffle = 200 trial-shuffled correlation coefficients r ′ ij were computed, then a p-value was determined as the fraction of shuffled coefficients r ′ ij whose absolute value exceeded the absolute value of the experimental correlation, p = # r ′ ij <sup>&</sup>gt;|rij<sup>|</sup> Nshuffle . For example, a correlation r was significant at p = 0.05 confidence level if no more than 10 shuffled correlation coefficients out of 200 exceeded r.

The pair-wise correlations of firing rates vectors computed in bins of fixed duration T were given by Equation (14) with ni b,s replaced by n<sup>i</sup> b,s /T. Instead, correlations of firing rates vectors inside hidden states (which have variable duration) were estimated after replacing n<sup>i</sup> b,s in Equation (14) with ν<sup>i</sup> (m,s), the firing rate of neuron i in state m in trial s. For each trial s, this quantity was computed according to Equation (1).

## Estimation of Dimensionality

The eigenvalues λ<sup>j</sup> in Equation (2) were found with a standard Principal Component Analysis (PCA) of the set of all firing rate vectors (Chapin and Nicolelis, 1999). The firing rate vectors were obtained via the HMM analysis (see Equation 1); all data from either ongoing or evoked activity were used. For the analysis of **Figure 3E**, where the duration and number of trials were varied, only the firing rate vectors of the HMM states present in the given trial snippet were used (even if present for only a few ms). When firing rate vectors in hidden states were not available (mainly, in "shuffled" datasets and in asynchronous homogeneous networks, see below for details), the firing rates were computed as spike counts in T = 200 ms bins divided by T, n<sup>i</sup> b,s /T, where ni b,s is as defined in Equation (14) (**Figures 3F,G**, **6E**, **7D**, **9A**). Dimensionality values were averaged across 20 simulated sessions for each ensemble size N; in each session, 40 trials of 5 s duration, resulting in N<sup>T</sup> = 1,000 bins, were used (using bin widths of 50– 500 ms did not change the results). Note that for the purpose of computing the dimensionality (Equation 3), it is equivalent to use either the binned firing rate n<sup>i</sup> b,s /T or the spike count ni b,s .

In our data, d roughly corresponded to the number of principal components explaining between 80 and 90% of the variance. However, note that all eigenvalues are retained in our definition of dimensionality given in Equation (2) above.

#### Shuffled Datasets

The dimensionality of the data as a function of ensemble size N was validated against surrogate datasets constructed by shuffling neurons across different sessions while matching the empirical distribution of ensemble sizes. Comparison analyses between empirical and shuffled ensembles were trial-matched using the minimal number of trials per condition across ensembles, and then tested for significant difference with the Mann-Whitney test on samples obtained from 20 bootstrapped ensembles. Neurons whose firing rate variance exceeded the population average by two standard deviations were excluded (8/167 of non-silent, non-somatosensory neurons).

#### Dependence on the Number of Trials: Simulations (Figures 7E, 8A)

The estimate of d from data depends on the number and duration of the trials (**Figure 3E** and Equation 16 below). To investigate this phenomenon in a simple numerical setting we generated N × N<sup>T</sup> "nominal" firing rates, thought of as originating from N neurons, each sampled N<sup>T</sup> times (trials). The single firing rates were sampled according to a log-normal distribution with equal means and covariance leading to Equation (7), i.e., Cij = ρσiσj(1 − δij) + σ 2 i δij, with δij = 1 if i = j, and zero otherwise (note that the actual distribution used is immaterial since the dimensionality only depends on the covariance matrix, see Equation 3). We considered the two cases of equal variance for all ensemble neurons, σ<sup>i</sup> = σ for all i (**Figure 8A**) or variances σ<sup>i</sup> sampled from a log-normal distribution (**Figure 8A** and "+" in **Figure 7E**). The same N and N<sup>T</sup> as used for the analysis of the model simulations in **Figure 7D** were used (where the "trials" were N<sup>T</sup> bins of 200 ms in 40 intervals of 5 second duration for each ensemble size N). The covariance of the data thus generated was estimated according to Equation (14), based on which the dimensionality Equation (3) was computed. The estimated dimensionality depends on N and N<sup>T</sup> and was averaged across 100 values of d, each obtained as explained above. Note that in this simplified setting increasing the duration of each trial is equivalent to adding more trials, i.e., the effect of having a trial 400 ms long producing 2 firing rates (one for each 200 ms bin) is equivalent to having two trials of 200 ms duration. In the general case, the effect of trial duration on d will depend on how trial duration affects the variance and correlations of the firing rates.

#### Dependence on the Number of Trials: Theory

The dependence of dimensionality on the number of trials can be computed analytically under the assumption that N ensemble neurons generate spike counts n<sup>i</sup> , for i = 1, . . . , N, distributed according to a multivariate Gaussian. Since we are interested in the spike-count covariance Equation (14), we can assume the spike-count distribution to have zero mean and true covariance Cij. The matrix M(NT) = (N<sup>T</sup> − 1) · S (NT) , where S (NT) is the covariance matrix Equation (14) sampled from N<sup>T</sup> trials, is distributed according to a Wishart distribution W<sup>N</sup> Cij, N<sup>T</sup> − 1 with N<sup>T</sup> − 1 degrees of freedom (Mardia et al., 1979). Since the variance of the Wishart distribution,

$$\operatorname{Var}\left(M\_{i\bar{j}}\right) = (N\_T - 1)(C\_{i\bar{j}}^2 + C\_{i\bar{i}}C\_{j\bar{j}}),$$

is proportional to NT, we obtain the variance of the entries of the sample covariance as

$$\operatorname{Var}\left(\mathcal{S}\_{ij}^{(N\_T)}\right) = \frac{\mathcal{C}\_{ij}^2 + \mathcal{C}\_{ii}\mathcal{C}\_{jj}}{N\_T - 1},\tag{15}$$

,

4 ,

to be used in the estimator of d (from Equation 3)

$$
\widehat{d} = \frac{[Tr(S)]^2}{Tr(S^2)} = \frac{\bar{b}\_N^{2^-}}{\widehat{c\_N} + \widehat{a\_N}},
$$

where <sup>a</sup>cN, <sup>c</sup>bN, <sup>b</sup>c<sup>2</sup> N are given by Equation (4) with C replaced by S. With a calculation similar to that used to obtain Equation (9), to leading order in N and N<sup>T</sup> one finds

$$E[\hat{d}] \approx \frac{E[\bar{b}\_N^2]}{E[\hat{c}\_N] + E[\widehat{a\_N}]},$$

with

$$E[\widehat{a}\widehat{N}] = N\left(\sigma^4 + 8\sigma^4\right) + \frac{2N\sigma^4}{N\_T - 1}$$

$$\begin{aligned} E[\widehat{b\_N^2}] &= N^2 \sigma^4 + N 8 \sigma^4 + \frac{2N\sigma^4}{N\_T - 1}, \\\\ E\left[\widehat{c\_N}\right] &= \left(N^2 - N\right) \left(\rho^2 + 8\rho^2\right) \sigma^4 + \left(N^2 - N\right) \frac{1 + \rho^2 + 8\rho^2}{N\_T - 1} \sigma^4 \end{aligned}$$

where we also used Equations (10) and (11), with Var- σ 2 i = δσ 4 and Var- rij = δρ 2 , for i 6= j. In conclusion, one finds

$$E[\hat{d}] = \frac{\left(N^2 + \frac{2N}{N\_T - 1}\right)\sigma^4 + N8\sigma^4}{\left(N^2 - N\right)\left(\rho^2 + 8\rho^2 + \frac{1 + \rho^2 + 8\rho^2}{N\_T - 1}\right)\sigma^4 + N\left(1 + \frac{2}{N\_T - 1}\right)\sigma^4 + N8\sigma^4}.\tag{16}$$

#### Model Fitting

The dependence of the data's dimensionality on ensemble size N was fitted by a straight line via standard least-squares,

$$d = \beta\_1 \cdot N + \beta\_0,$$

separately for ongoing and evoked activity (**Figures 3B–D**, **6B–D**). Comparison between the dimensionality of evoked and ongoing activity was carried out with a 2-way ANOVA with condition (evoked vs. ongoing) and ensemble size (N) as factors. Since d depends on the number and duration of the trials used to estimate the covariance matrix (**Figure 3E** and Equation 16), we matched both the number of trials and trial length in comparisons of ongoing and evoked dimensionality. If multiple tastes were used, the evoked trials were each matched to a random subset of an equal number of ongoing trials.

The dependence of dimensionality d on ensemble size N in a surrogate dataset of Poisson spike trains with mean pairwise correlation ρ (generated according to the algorithm described in the next section) was modeled as Equation (16) with δρ <sup>2</sup> = αρ 2 and δσ <sup>4</sup> = σ <sup>4</sup> = β (**Figure 7D**, dashed lines); N<sup>T</sup> was fixed to 1000 (40 trials of 5 s each, segmented in 200 ms bins). The parameters α, β were tuned to fit all Poisson trains simultaneously on datasets with N = 5, 10, . . . , 100 and ρ = 0, 0.01, 0.05, 0.1, 0.2, with 20 ensembles for each value (**Figure 7D**; only the fits for ρ = 0, 0.1, 0.2 are shown). A standard non-linear least-squares procedure was used (Holland and Welsch, 1977).

# Generation of Correlated Poisson Spike Trains

Ensembles of independent and correlated Poisson spike trains were generated for the analysis of **Figure 7**. Ensembles of independent stationary Poisson spike trains with given firing rates ν<sup>i</sup> were generated by producing their interspike intervals according to an exponential distribution with parameter ν<sup>i</sup> . Stationary Poisson spike trains with fixed pairwise correlations (but no temporal correlations) were generated according to the method reported in Macke et al. (2009), that we briefly outline below.

We split each trial into 1 ms bins and consider the associated binary random variable X<sup>i</sup> (t) = 1 if the i-th neuron emitted a spike in the t-th bin, and X<sup>i</sup> (t) = 0 if no spike was emitted. These samples were obtained by first drawing a sample from an auxiliary N-dimensional Gaussian random variable U ∼ N (γ, 3) and then thresholding it into 0 and 1: X<sup>i</sup> = 1 if U<sup>i</sup> > 0, and X<sup>i</sup> = 0 otherwise. Here, γ = {γ1, γ2, . . . , γN} is the mean vector and 3 = 3ij is the covariance matrix of the N-dimensional Gaussian variable U. For appropriately chosen

parameters γ<sup>i</sup> and 3ij the method generates correlated spike trains with the desired firing rates ν<sup>i</sup> and pairwise spike count correlation coefficients rij.

The prescription for γ<sup>i</sup> and 3ij is most easily expressed as a function of the desired probabilities µ<sup>i</sup> of having a spike in a bin of width dt, µ<sup>i</sup> = P(X<sup>i</sup> (t) = 1), and the pairwise covariance cij of the random binary vectors Xi(t) and Xj(t), from which γ<sup>i</sup> and 3ij can be obtained by inverting the following relationships:

$$\begin{aligned} \mu\_i &= \Phi \left( \boldsymbol{\gamma}\_i \right), \\ c\_{ii} &= \Phi \left( \boldsymbol{\gamma}\_i \right) \Phi \left( -\boldsymbol{\gamma}\_i \right), \\ c\_{ij} &= \Phi\_2 \left( \boldsymbol{\gamma}\_i, \boldsymbol{\gamma}\_j, \Lambda\_{ij} \right) - \Phi \left( \boldsymbol{\gamma}\_i \right) \Phi \left( \boldsymbol{\gamma}\_j \right), i \neq j. \end{aligned}$$

Here, 8 (x) is the cumulative distribution of a univariate Gaussian with mean 0 and variance 1 evaluated at x, and 8<sup>2</sup> x, y, 3 is the cumulative distribution of a bivariate Gaussian with means 0, variances 1 and covariance 3 evaluated at (x, y) (note that the distributions 8 and 8<sup>2</sup> are unrelated to the Ndimensional Gaussian U ∼ N (γ, 3)). Without loss of generality we imposed unit variances for U<sup>i</sup> , i.e., 3ii = 1.

We related the spike probabilities µ<sup>i</sup> to the firing rates ν<sup>i</sup> as µ<sup>i</sup> = 1 − e <sup>−</sup>νidt, with (1 − µi) being the probability of no spikes in the same bin. When dt approaches zero, µ<sup>i</sup> ≈ νidt and the spike trains generated as vectors of binary random variables by sampling U ∼ N (γ, 3) will approximate Poisson spike trains (dt = 1 ms bins were used). In order to have a fair comparison with the data generated by the spiking network model (described in the next section), the mean firing rates of the Poisson spike trains were matched to the average firing rates obtained from the simulated data.

Since γ and 3 were the same in all bins, values of X<sup>i</sup> (t) and X<sup>i</sup> (s) were independent for t 6= s (i.e., the spike trains had no temporal correlations). As a consequence, the random binary vectors have the same pair-wise correlations as the spike counts, and the cij are related to the desired rij by cij = <sup>r</sup>ij<sup>q</sup> µ<sup>i</sup> (1 − µi)µ<sup>j</sup> 1 − µ<sup>j</sup> , where µ<sup>i</sup> (1 − µi) is the variance of Xi . See Macke et al. (2009) for further details.

# Spiking Network Model

We modeled the data with a recurrent spiking network of N = 5000 randomly connected leaky integrate-and-fire (LIF) neurons, of which 4000 excitatory (E) and 1000 inhibitory (I). Connection probability pβ<sup>α</sup> from neurons in population α ∈ E,I to neurons in population β ∈ E,I were pEE = 0.2 and pEI = pIE = pII = 0.5; a fraction f = 0.9 of excitatory neurons were arranged into Q different clusters, with the remaining neurons belonging to an unstructured ("background") population (Amit and Brunel, 1997). Synaptic weights Jβ<sup>α</sup> from neurons in population α ∈ E,I to neurons in population β ∈ E,I scaled with N as Jβ<sup>α</sup> = jβα/ √ N, with jβα constants having the following values (units of mV): jEI = 3.18, jIE = 1.06, jII = 4.24, jEE = 1.77. Within an excitatory cluster synaptic weights were potentiated, i.e., they took average values of hJi<sup>+</sup> = J<sup>+</sup> jEE with J<sup>+</sup> > 1, while synaptic weights between units belonging to different clusters were depressed to average values hJi<sup>−</sup> = J<sup>−</sup> jEE, with J<sup>−</sup> = 1 − γf (J<sup>+</sup> − 1) < 1, with γ = 0.5. The latter relationship between J+ and J− helps to maintain balance between overall potentiation and depression in the network (Amit and Brunel, 1997).

Below spike threshold, the membrane potential V of each LIF neuron evolved according to

$$\mathfrak{r}\_m \frac{dV}{dt} = -V + \mathfrak{r}\_m \left( I\_{rec} + I\_{ext} + I\_{stin} \right)$$

with a membrane time constant τ<sup>m</sup> = 20 ms for excitatory and 10 ms for inhibitory units. The input current was the sum of a recurrent input Irec, an external current Iext representing an ongoing afferent input from other areas, and an external stimulus Istim representing e.g., a delivered taste during evoked activity only. In our units, a membrane capacitance of 1nF is set to 1. A spike was said to be emitted when V crossed a threshold Vthr, after which V was reset to a potential Vreset = 0 for a refractory period of τref = 5 ms. Spike thresholds were chosen so that, in the unstructured network (i.e., with J<sup>+</sup> = J<sup>−</sup> = 1), the E and I populations had average firing rates of 3 and 5 spikes/s, respectively (Amit and Brunel, 1997). The recurrent synaptic input I i rec to unit i evolved according to the dynamical equation

$$\text{tr}\_s \frac{dI\_{\text{rec}}^i}{dt} = -I\_{\text{rec}}^i + \sum\_{j=1}^N J\_{\vec{\eta}} \sum\_k \\$\left(t - \mathbf{r}\_k^j\right),$$

where t j k was the arrival time of k-th spike from the j-th presynaptic unit, and τ<sup>s</sup> was the synaptic time constant (3 and 2 ms for E and I units, respectively), resulting in an exponential postsynaptic current in response to a single spike, <sup>J</sup>ij τs exp(−t/τs)2(t), where 2 (t) = 1 for t ≥ 0, and 2 (t) = 0 otherwise. The ongoing external current to a neuron in population α was constant and given by Iext = Nextpα0Jα0νext, where Next = nEN, pα<sup>0</sup> = pEE, Jα<sup>0</sup> = <sup>√</sup> jα0 N with jE<sup>0</sup> = 0.3, jI<sup>0</sup> = 0.1, and νext = 7 spikes/s. During evoked activity, stimulus-selective units received an additional input representing one of the four incoming stimuli. The stimuli targeted combinations of neurons as observed in the data. Specifically, the fractions of neurons responsive to n = 1, 2, 3 or all 4 stimuli were 17% (27/162), 22% (36/162), 26% (42/162), and 35% (57/162) (Jezzini et al., 2013; Mazzucato et al., 2015). Each stimulus had constant amplitude νstim ranging from 0 to 0.5 νext. In the following we measure the stimulus amplitude as percentage of νext (e.g., "10%" corresponds to νstim = 0.1 νext). The onset of each stimulus was always t = 0, the time of taste delivery. The stimulus current to a unit in population α was constant and given by Istim = Nextpα0Jα0νstim.

# Mean Field Analysis of the Model

The stationary states of the spiking network model in the limit of large N were found with a mean field analysis (Amit and Brunel, 1997; Brunel and Hakim, 1999; Fusi and Mattia, 1999; Curti et al., 2004; Mazzucato et al., 2015). Under typical conditions, each neuron of the network receives a large number of small postsynaptic currents (PSCs) per integration time constant. In such a case, the dynamics of the network can be analyzed under the diffusion approximation within the population density approach. The network has α = 1, . . . , Q + 2 sub-populations, where the first Q indices label the Q excitatory clusters, α = Q + 1 labels the "background" units, and α = Q + 2 labels the homogeneous inhibitory population. In the diffusion approximation (Tuckwell, 1988; Lánský and Sato, 1999; Richardson, 2004), the input to each neuron is completely characterized by the infinitesimal mean µ<sup>α</sup> and variance σ 2 α of the post-synaptic potential (see Mazzucato et al., 2015 for the expressions of the infinitesimal mean and variance for all subpopulations).

Parameters were chosen so that the network with J<sup>+</sup> = J<sup>−</sup> = 1 (where all E → E synaptic weights are equal) would operate in the balanced asynchronous regime (van Vreeswijk and Sompolinsky, 1996, 1998; Renart et al., 2010), where incoming contributions from excitatory and inhibitory inputs balance out, neurons fire irregular spike trains with weak pair-wise correlations.

The unstructured network has only one dynamical state, i.e., a stationary point of activity where all E and I neurons have constant firing rate ν<sup>E</sup> and ν<sup>I</sup> , respectively. In the structured network (where J+ > 1), the network undergoes continuous transitions among a repertoire of states, as shown in the main text. To avoid confusion between network activity states and HMM states, we refer to the former as network "configurations" instead of states. Admissible networks configurations must satisfy the Q + 2 self-consistent mean field equations (Amit and Brunel, 1997)

$$\upsilon\_{\alpha} = F\_{\alpha} \left( \mu\_{\alpha} (\overrightarrow{\vee} \,), \sigma\_{\alpha}^{2} (\overrightarrow{\vee} \,) \right),$$

where −→<sup>ν</sup> <sup>=</sup> h ν1, . . . , νQ, ν bg E , ν<sup>I</sup> i is the firing rate vector and Fα µα, σ<sup>2</sup> α is the current-to-rate response function of the LIF neurons. For fast synaptic times, i.e., <sup>τ</sup><sup>s</sup> <sup>τ</sup><sup>m</sup> ≪ 1, F<sup>α</sup> µα, σ<sup>2</sup> α is well approximated by (Brunel and Sergi, 1998; Fourcaud and Brunel, 2002)

$$F\_{\mathfrak{a}}\left(\mu\_{\mathfrak{a}},\sigma\_{\mathfrak{a}}\right) = \left(\mathfrak{r}\_{\mathfrak{r}\mathfrak{f}} + \mathfrak{r}\_{m,\mathfrak{a}}\sqrt{\pi} \int\_{H\_{\mathfrak{c}\mathfrak{f}',\mathfrak{a}}}^{\Theta\_{\mathfrak{c}\mathfrak{f}',\mathfrak{a}}} e^{\mathfrak{a}^2} \left[1 + \text{erf}\left(\mathfrak{u}\right)\right] \right)^{-1},$$

where

$$
\Theta\_{\it eff,\alpha} = \frac{V\_{\it llr,\alpha} - \mu\_{\alpha}}{\sigma\_{\alpha}} + a k\_{\alpha},
$$

$$H\_{\mathfrak{eff},\alpha} = \frac{V\_{reset,\alpha} - \mu\_{\alpha}}{\sigma\_{\alpha}} + a k\_{\alpha} \frac{1}{r}$$

where k<sup>α</sup> = p τs,α/τm,<sup>α</sup> is the square root of the ratio of synaptic time constant to membrane time constant, and a = |ζ(1/2)| √ 2 ∼ 1.03. This theoretical response function has been fitted successfully to the firing rate of neocortical neurons in the presence of in vivo-like fluctuations (Rauch et al., 2003; Giugliano et al., 2004; La Camera et al., 2006, 2008).

The fixed points −→<sup>ν</sup> ∗ of the mean field equations were found with Newton's method (Press et al., 2007). The fixed points can be either stable (attractors) or unstable depending on the eigenvalues λ<sup>α</sup> of the stability matrix

$$\begin{split} S\_{\alpha\beta} &= \frac{1}{\mathfrak{r}\_{s,\alpha}} \left( \frac{\partial F\_{\alpha}\left(\mu\_{\alpha}(\overrightarrow{\mathbb{V}^{\flat}}), \sigma\_{\alpha}^{2}(\overrightarrow{\mathbb{V}^{\flat}})\right)}{\partial \upsilon\_{\beta}} \right) \\ &- \left( \frac{\partial F\_{\alpha}\left(\mu\_{\alpha}(\overrightarrow{\mathbb{V}^{\flat}}), \sigma\_{\alpha}^{2}(\overrightarrow{\mathbb{V}^{\flat}})\right)}{\partial \sigma\_{\alpha}^{2}} \frac{\partial \sigma\_{\alpha}^{2}}{\partial \upsilon\_{\beta}} - \delta\_{\alpha\beta} \right), \end{split}$$

evaluated at the fixed point −→<sup>ν</sup> ∗ (Mascaro and Amit, 1999). If all eigenvalues have negative real part, the fixed point is stable (attractor). If at least one eigenvalue has positive real part, the fixed point is unstable. Stability is meant with respect to an approximate linearized dynamics of the mean and variance of the input current:

$$\begin{aligned} \mathfrak{r}\_{s,\alpha} \frac{dm\_{\alpha}}{dt} &= -m\_{\alpha} + \mu\_{\alpha}(\overrightarrow{\nabla}) \\\\ \frac{\mathfrak{r}\_{s,\alpha}}{2} \frac{ds\_{\alpha}^{2}}{dt} &= -s\_{\alpha}^{2} + \sigma\_{\alpha}^{2}(\overrightarrow{\nabla}) \\\\ \alpha\_{\alpha}(t) &= F\_{\alpha}\left(m\_{\alpha}(\overrightarrow{\nabla}), s\_{\alpha}^{2}(\overrightarrow{\nabla})\right) \end{aligned}$$

mα(

α ( ,

ν<sup>α</sup> (t) = F<sup>α</sup>

where µ<sup>α</sup> and σ 2 α are the stationary values for fixed −→<sup>ν</sup> given earlier. For fast synaptic dynamics in the asynchronous balanced regime, these rate dynamics are in very good agreement with simulations (La Camera et al., 2004—see Renart et al., 2004; Giugliano et al., 2008 for more detailed discussions).

# Metastable Configurations in the Network Model

The stable configurations of a network with an infinite number of neurons were obtained in the mean field approximation of the previous section and are shown in **Figure 4B** for Q = 30 and a range of values of the relative potentiation parameter J+. Above the critical point J<sup>+</sup> = 4.2, stable configurations characterized by a finite number of active clusters emerge (gray lines; the number of active clusters is reported next to each line). For a given J+, the firing rate is the same in all active clusters and is inversely proportional to the total number of active clusters. Stable patterns of firing rates are also found in the inhibitory population (red lines), in the inactive clusters (having low firing rates; gray dashed lines), and in the unstructured excitatory population (dashed blue lines). For a fixed value of J+, multiple stable configurations coexist with different numbers of active clusters. For example, for J<sup>+</sup> = 5.3, configurations with up to 7 active clusters are stable, each configuration with different firing rates. This generates multistable firing rates in single neurons, i.e., the property, also observed in the data, that single neurons can attain more than 2 firing rates across states (Mazzucato et al., 2015). Note that if J<sup>+</sup> ≤ 5.15 an alternative stable configuration of the network with all clusters inactive (firing rates < 10 spikes/s) is also possible (single brown line).

Strictly speaking, the configurations in **Figure 4B** are stable only in a network containing an infinite number of uncorrelated neurons. In a finite network (or when neurons are strongly correlated) these configurations can lose stability due to strong fluctuations, which ignite transitions among the different configurations. Full details are reported in Mazzucato et al. (2015).

# Model Simulations and Analysis of Simulated Data

The dynamical equations of the LIF neurons were integrated with the Euler algorithm with a time step of dt = 0.1 ms. We simulated 20 different networks (referred to as "sessions" in the following) during both ongoing and evoked activity. We chose four different stimuli per session during evoked activity (to mimic taste delivery). Trials were 5 s long. The HMM analyses for **Figures 2**, **5** were performed on ensembles of randomly selected excitatory neurons with the same procedure used for the data (see previous section "Hidden Markov Model (HMM) analysis"). The ensemble sizes were chosen so as to match the empirical ensemble sizes (3–9 randomly selected neurons). For the analysis of **Figure 9A**, ensembles of increasing size (from 5 to 100 neurons) were used from simulations with Q = 30 clusters. When the ensemble size was less than the number of clusters (N ≤ Q), each neuron was selected randomly from a different cluster; when ensemble size was larger than the number of clusters, one neuron was added to each cluster until all clusters were represented, and so on until all N neurons had been chosen. To allow comparison with surrogate Poisson spike trains, the dimensionality of the simulated data was computed from the firing rate vectors in T = 200 ms bins as explained in section "Dimensionality measure." For control, the dimensionality was also computed from the firing rate vectors in hidden states obtained from an HMM analysis, obtaining qualitatively similar results.

# RESULTS

# Dimensionality of the Neural Activity

We investigate the dimensionality of sequences of firing rate vectors generated in the GC of alert rats during periods of ongoing or evoked activity (see Methods). To provide an intuitive picture of the meaning of dimensionality adopted in this paper, consider the firing rate vectors from N simultaneously recorded neurons. These vectors can occupy, a priori, the entire Ndimensional vector space minimally required to describe the population activity of N independent neurons. However, the sequence of firing rate vectors generated by the neural dynamics may occupy a subspace that is spanned by a smaller number m < N of coordinate axes. For example, the data obtained by the ensemble of three simulated spike counts in **Figure 1** mostly lie on a 2D space, the plane shaded in gray. Although 3 coordinates are still required to specify all data points, a reduced representation of the data, such as that obtained from PCA, would quantify the dimension of the relevant subspace as being close to 2. To quantify this fact we use the following definition of dimensionality (Abbott et al., 2011)

$$d = \left(\sum\_{i=1}^{N} \widetilde{\lambda}\_i^2\right)^{-1}.$$

where <sup>N</sup> is the ensemble size and <sup>e</sup>λ<sup>i</sup> are the normalized eigenvalues of the covariance matrix, each expressing the fraction of the variance explained by the corresponding principal component (see Methods for details). According to this formula, if the first n eigenvalues express each a fraction 1/n of the variance while the remaining eigenvalues vanish, the dimensionality is d = n. In less symmetric situations, d reflects roughly the dimension of the linear subspace explaining most variance about all data points. In the example of the data on the gray plane of **Figure 1**, d = 1.8, which is close to 2, as expected. Similarly, data points lying mostly along the blue and red straight lines in **Figure 1** have a dimensionality of 0.9, close to 1. In all cases, d > 0 and d ≤ N, where N is the ensemble size.

The blue and red data points in **Figure 1** were obtained from a fictitious scenario where neuron 1 and neuron 2 were selective to surrogate stimuli A and B, respectively, and are meant to mimic two possible evoked responses. The subspace containing responses to both stimuli A and B would have a dimensionality dA+<sup>B</sup> = 1.7, similar to the dimensionality of the data points distributed on the gray plane (meant instead to represent spike counts during ongoing activity in the same fictitious scenario). Thus, a dimensionality close to 2 could originate from different patterns of activity, such as occupying a plane or two straight lines. Other and more complex scenarios are, of course, possible. In general, the dimensionality

will reflect existing functional relationships among ensemble neurons (such as pair-wise correlations) as well as the response properties of the same neurons to external stimuli. The pictorial example of **Figure 1** caricatures a stimulus-induced reduction of dimensionality, as found in the activity of simultaneously recorded neurons from the GC of alert rats, as we show next.

# Dimensionality is Proportional to Ensemble Size

We computed the dimensionality of the neural activity of ensembles of 3–9 simultaneously recorded neurons in the gustatory cortex of alert rats during the 5 s inter-trial period preceding (ongoing activity) and following (evoked activity) the delivery of a taste stimulus (said to occur at time t = 0; see Methods). Ensemble activity in single trials during both ongoing (**Figure 2A**) and evoked activity (**Figure 2B**) could be characterized in terms of sequences of metastable states, where each state is defined as a collection of firing rates across simultaneously recorded neurons (Jones et al., 2007; Mazzucato et al., 2015). Transitions between consecutive states were detected via a Hidden Markov Model (HMM) analysis, which provides the probability that the network is in a certain state at every 1 ms bin (**Figure 2**, color-coded lines superimposed to raster plots). The ensemble of spike trains was considered to be in a given state if the posterior probability of being in that state exceeded 80% in at least 50 consecutive 1-ms bins (**Figure 2**, color-coded shaded areas). Transitions among states were triggered by the co-modulation of a variable number of ensemble neurons and occurred at seemingly random times (Mazzucato et al., 2015). For this reason, the dimensionality of the neural activity was computed based on the firing rate vectors in each HMM state (one firing rate vector per state per trial; see Methods for details).

The average dimensionality of ongoing activity across sessions was dongoing = 2.6 ± 1.2 (mean±SD; range: [1.2, 5.0]; 27 sessions). An example of the eigenvalues for a representative ensemble of eight neurons is shown in **Figure 3A**, where d = 4.42. The dimensionality of ongoing activity was approximately linearly related to ensemble size (**Figure 3B**, linear regression, r = 0.4, slope bongoing = 0.26 ± 0.12, p = 0.04). During evoked activity dimensionality did not differ across stimuli (oneway ANOVA, no significant difference across tastants, p > 0.8), hence all evoked data points were combined for further analysis. An example of the eigenvalue distribution of the ensemble in **Figure 2B** is shown in **Figure 3C**, where devoked = 1.3 ∼ 1.7 across 4 different taste stimuli. Across all sessions, dimensionality was overall smaller (devoked = 2.0 ± 0.6, mean±SD, range: [1.1, 3.9]) and had a reduced slope as a function of N compared to ongoing activity (**Figure 3D**, linear regression, r = 0.39, slope bevoked = 0.13 ± 0.03, p < 10−<sup>4</sup> ). However, since dimensionality depends on the number and duration of the trials used for its estimation (**Figure 3E**), a proper comparison requires matching trial number and duration for each data point, as described next.

# Stimulus-Induced Reduction of Dimensionality

We matched the number and duration of the trials for each data point and ran a two-way ANOVA with condition (ongoing vs. evoked) and ensemble size as factors. Both the main dimensionality [F(1, 202) = 11.93, p < 0.001] and the slope were significantly smaller during evoked activity [test of interaction, F(6, <sup>202</sup>) = 5.09, p < 10−<sup>4</sup> ]. There was also a significant effect of ensemble size [F(6, 202) = 18.72, p < 10−14], confirming the results obtained with the separate regression analyses. These results suggest that stimuli induce a reduction of the effective space visited by the firing rate vector during evoked activity. This was confirmed by a paired sample analysis of the individual dimensionalities across all 27 × 4=108 ensembles (27 ensemble times 4 gustatory stimuli; p < 0.002, Wilcoxon signed-rank test).

# Dimensionality is larger in Ensembles of Independent Neurons

The dimensionality depends on the pair-wise correlations of simultaneously recorded neurons. Shuffling neurons across ensembles would destroy the correlations (beyond those expected by chance), and would give a measure of how different the dimensionality of our datasets would be compared to sets of independent neurons. We measured the dimensionality of surrogate datasets obtained by shuffling neurons across sessions; because shuffling destroys the structure of the hidden states, firing rates in bins of fixed duration (200 ms) were used to estimate the dimensionality (see Methods for details). As expected, the slope of d vs. N was larger in the shuffled datasets compared to the simultaneously recorded ensembles (not shown) during both ongoing activity (bshuff = 0.67 ± 0.06 vs. bdata = 0.60 ± 0.01; mean ± SD, Mann-Whitney test, p < 0.001, 20 bootstraps), and evoked activity (bshuff = 0.36 ± 0.07 vs. bdata = 0.29 ± 0.01; p < 0.001). Especially during ongoing activity, this result was accompanied by a narrower distribution of pair-wise correlations in the shuffled datasets compared to the simultaneously recorded datasets (**Figure 3G**), and is consistent with an inverse relationship between dimensionality and pair-wise correlations (see Equation 9).

# Time Course of Dimensionality as a Function of Ensemble Size

Unlike ongoing activity, the dependence of dimensionality on ensemble size (the slope of the linear regression of d vs. N) was modulated during different epochs of the post-stimulus period [**Figure 3F**, full lines; two-way ANOVA; main effect of time F(4, 495) = 3.80, p < 0.005; interaction time x condition: F(4, <sup>495</sup>) = 4.76, p < 0.001]. In particular, the dependence of d on the ensemble size N almost disappeared immediately after stimulus presentation in the simultaneously recorded, but not in the shuffled ensembles (trial-matched slope in the first evoked second: bevoked = 0.07 ± 0.01 vs bshuff = 0.19 ± 0.07) and converged to a stable value after approximately 1 second (slope after the first second bevoked = 0.38 ± 0.01; compare with a stable average slope during ongoing activity of bongoing = 0.57 ± 0.01, **Figure 3F**).

Note that the dimensionality is larger when the firing rate is computed in bins (as in **Figure 3F**) rather than in HMM states (as in **Figures 3B–D**, where the slopes are about half than in **Figure 3F**). The reason is that firing rates and correlations are

approximately constant during the same HMM state, whereas they may change when estimated in bins of fixed duration that include transitions among hidden states. These changes tend to dilute the correlations resulting in higher dimensionality as predicted e.g., by Equation (9). A comparison of the pair-wise correlations of binned firing rates (**Figure 3G**) vs. those of firing rates in HMM states (**Figure 3H**) confirmed this hypothesis. Also, if the argument above is correct, one would expect a dependence of dimensionality on (fixed) bin duration. We computed the correlations and dimensionality of binned firing rates for various bin durations and found that r increases and d decreases for increasing bin durations (not shown). However, the slope of d vs. N is always larger in ongoing than in evoked activity regardless of bin size (ranging from 10 ms to 5 s; not shown). This confirms the generality of the results of **Figures 3B–D**, which were obtained using firing rate vectors in hidden states.

To summarize our main results so far, we found that dimensionality depends on ensemble size during both ongoing and evoked activity, and such dependence is significantly reduced in the post-stimulus period. This suggests that while state sequences during ongoing activity explore a large portion of the available firing rate space, the presentation of a stimulus initially collapses the state sequence along a more stereotyped and lower-dimensional response (Katz et al., 2001; Jezzini et al., 2013). During both ongoing and evoked activity, the dimensionality is also different than expected by chance in a set of independent neurons (shuffled datasets).

# Clustered Spiking Network Model of Dimensionality

To gain a mechanistic understanding of the different dimensionality of ongoing and evoked activity, we have analyzed a spiking network model with clustered connectivity which has been shown to capture many essential features of the data (Mazzucato et al., 2015). In particular, the model reproduces the transitions among latent states in both ongoing and evoked activity. The network (see Methods for details) comprises Q clusters of excitatory neurons characterized by stronger synaptic connections within each cluster and weaker connections between neurons in different clusters. All neurons receive recurrent input from a pool of inhibitory neurons that keeps the network in a balanced regime of excitation and inhibition in the absence of external stimulation (**Figure 4A**). In very large networks (technically, in networks with an infinite number of neurons), the stable configurations of the neural activity are characterized by a finite number of active clusters whose firing rates depend on the number clusters active at any given moment, as shown

FIGURE 3 | Dependence of dimensionality on ensemble size (data). (A) Fraction of variance explained by each principal eigenvalue for an ensemble of 8 neurons during ongoing activity (corresponding to the filled dot in panel B) in the empirical dataset. The dashed vertical line represents the value of the dimensionality for this ensemble (d = 4.4). X-axis, eigenvalue number; Y-axis, fraction of variance explained by each eigenvalue. (B) Dimensionality of neural activity across all ensembles in the empirical dataset during ongoing activity (circles, linear regression fit, d = b · N + a, b = 0.26 ± 0.12, a = 1.07 ± 0.74, r = 0.4), estimated from HMM firing rate fits on all ongoing trials in each session (varying from 73 to 129). X-axis: ensemble size; Y-axis: dimensionality. (C) Fraction of variance explained by each principal eigenvalue for the ensemble in (A) during evoked activity. Principal eigenvalues for sucrose (S, orange), sodium chloride (N, yellow), citric acid (C, cyan), and quinine (Q, blue) are presented (corresponding to the color-coded dots in panel D). X-axis, eigenvalue number; Y-axis, percentage of variance explained by each eigenvalue. (D) Dimensionality of neural activity across all ensembles in the empirical dataset during evoked activity (notations as in panel B, linear regression: d = b · N + a, b = 0.13 ± 0.03, a = 1.27 ± 0.19, r = 0.39), estimated from HMM firing rate fits on evoked trials in each condition (varying from 7 to 11 trials across sessions for each tastant). (E) The slope of the linear regression of dimensionality (d) vs. ensemble size (N) as a function of the length of the trial interval and the number of trials used to estimate the dimensionality. X-axis, length of trial interval [s]; Y-axis, number of trials. (F) Time course of the trial-matched slopes of d vs. N, evaluated with 200 ms bins in consecutive 1 s intervals during ongoing (black curve, t < 0) and evoked periods (red curve, t > 0; error bars represent SD). A significant time course is triggered by stimulus presentation (see Results for details). The slopes of the empirical dataset (thick curves) were smaller than the slope of the shuffled dataset (dashed curves) during ongoing activity. X-axis, time from stimulus onset at t = 0 [s]; Y-axis, slope of d vs. N. (G) Distribution of pair-wise correlations in simultaneously recorded ensembles (black and red histograms for ongoing and evoked activity, respectively) and shuffled ensembles (brown and pink dashed histograms for ongoing and evoked activity, respectively) from 200 ms bins. X-axis, correlation; Y-axis, frequency. (H) Distribution of pair-wise correlations from HMM states during ongoing (black) and evoked activity (red) for all simultaneously recorded pairs of neurons. X-axis, correlation; Y-axis, frequency.

in **Figure 4B** (where Q = 30). In a finite network, however, finite size effects ignite transitions among these configurations, inducing network states (firing rate vectors) on randomly chosen subsets of neurons that resemble the HMM states found in the data (**Figure 5**; see Mazzucato et al., 2015 for details).

The dimensionality of the simulated sequences during ongoing and evoked activity was computed as done for the data, finding similar results. For the examples in **Figure 5**, we found dongoing = 4.0 for ongoing activity (**Figure 6A**) between devoked = 2.2 and devoked = 3.2 across tastes during evoked activity (**Figure 6C**). Across all simulated sessions, we found an average dongoing = 2.9 ± 0.9 (mean ± SD) for ongoing activity and devoked = 2.4 ± 0.7 for evoked activity. The model captured the essential properties of dimensionality observed in the data: the dimensionality did not differ across different tastes (one-way ANOVA, p > 0.2) and depended on ensemble size during both ongoing (**Figure 6B**; slope = 0.36 ± 0.07, r = 0.77, p < 10−<sup>4</sup> ) and evoked periods (**Figure 6D**; slope = 0.12 ± 0.04, r = 0.29, p = 0.01). As for the data, the dependency on ensemble size was smaller for evoked compared to ongoing activity. We performed a trial-matched two-way ANOVA as done on the data and found, also in the model, a main effect of condition [ongoing vs. evoked: F(1, 146) = 22.1, p < 10−<sup>5</sup> ], a main effect of ensemble size [F(6, 146) = 14.1, p < 10−11], and a significant interaction [F(6, 146) = 3.8, p = 0.001]. These results were accompanied by patterns of correlations among the model neurons (**Figures 6E,F**) very similar to those found in the data (**Figures 3G,H**; see section "Dimensionality is larger in the presence of clusters" for statistics of correlation values). As in the data, narrower distributions of correlations were found for binned firing rates (**Figure 6E**) compared to firing rates in hidden states (**Figure 6F**; compare with **Figures 3G,H**, respectively). Moreover, shuffling neurons

FIGURE 4 | Recurrent network model. (A) Schematic recurrent network architecture. Triangles and squares represent excitatory and inhibitory LIF neurons respectively. Darker disks indicate excitatory clusters with potentiated intra-cluster synaptic weights. (B) Mean field solution of the recurrent network. Firing rates of the stable states for each subpopulation are shown as function of the intra-cluster synaptic potentiation parameter J+: firing rate activity in the active clusters (solid gray lines), firing rate in the inactive clusters (dashed gray lines), activity of the background excitatory population (dashed blue lines), activity of the inhibitory population (solid red lines). In each case, darker colors represent configurations with larger number of active clusters. Numbers denote how many clusters are active in each stable configuration. Configurations with 1–8 active clusters are stable in the limit of infinite network size. A global configuration where all clusters are inactive (brown line) becomes unstable at the value J+ = 5.15. The vertical green line represents the value of J+ = 5.3 chosen for the simulations. X-axis, intra-cluster potentiation parameter J+ in units of JEE ; Y-axis, Firing rate (spks/s).

across datasets reduced the correlations (**Figure 6E**, dashed), resulting in a larger slope of d vs. N (not shown). Finally, d during ongoing activity was always larger than during evoked activity also when computed on binned firing rates (not shown), as found in the data (see section "Dependence of dimensionality on bin size").

Since the model was not fine-tuned to find these results, the different dimensionalities of ongoing and evoked activity, and

regression: d = b · N + a, b = 0.12 ± 0.04, a = 1.70 ± 0.26, r = 0.29). (E) Distribution of pair-wise correlations in simultaneously recorded ensembles from the clustered network model (black and red histograms for ongoing and evoked activity, respectively) and in shuffled ensembles (brown and pink dashed histograms for ongoing and evoked activity, respectively) from 200 ms bins. X-axis, correlation; Y-axis, frequency. (F) Distribution of pair-wise correlations from HMM states during ongoing (black) and evoked activity (red) for all simultaneously recorded pairs of neurons. X-axis, correlation; Y-axis, frequency.

their associated patterns of pair-wise correlations, are likely the consequence of the organization in clusters and of the ensuing dynamics during ongoing and evoked activity.

# Scaling of Dimensionality with Ensemble Size and Pair-Wise Correlations

The dependence of dimensionality on ensemble size observed in the data (**Figure 3B**) and in the model (**Figure 6B**) raises the question of whether or not the dimensionality would converge to an upper bound as one increases the number of simultaneously recorded neurons. In general, this question is important in a number of settings, related e.g., to coding in motor cortex (Ganguli et al., 2008; Gao and Ganguli, 2015), performance in a discrimination task (Rigotti et al., 2013), or coding of visual stimuli (Cadieu et al., 2013). We can attack this question aided by the model of **Figure 4**, where we can study the effect of large numbers of neurons, but also the impact on dimensionality of a clustered network architecture compared to a homogeneous one, at parity of correlations and ensemble size.

We consider first the case of a homogeneous network of neurons having no clusters and low pair-wise correlations, but having the same firing rates distributions (which were approximately log-normal, **Figure 7A**) and the same mean pairwise correlations as found in the data (ρ ∼ 0.01 − 0.2). This would require solving a homogeneous recurrent network self-consistently for the desired firing rates and correlations. As a proxy for this scenario, we generated 20 sessions of 40 Poisson spike trains having exactly the desired properties (including the case of independent neurons for which ρ = 0). Two examples with ρ = 0 and ρ = 0.1, respectively, are shown in **Figures 7B,C**. Since in the asynchronous homogeneous network there are no transitions and hence no hidden states, the dimensionality was estimated based on the rate vectors in bins of 200 ms duration (using bin widths of 50–500 ms did not change the results; see Methods for details).

We found that the dimensionality grows linearly with ensemble size in the absence of correlations, but is a concave function of N in the presence of pair-wise correlations (circles in **Figure 7D**). Thus, as expected, the presence of correlations reduces the dimensionality and suggests the possibility of an upper bound. A simple theoretical calculation mimicking this scenario shows that d in this case converges indeed to an upper bound that depends on the inverse of the square of the pair-wise correlations. For example, in the case of uniform correlations (ρ) and equal variances of the spike counts, Equation (8) of Methods, d (N, ρ) = 1 ρ <sup>2</sup>+(1−ρ 2 )/N , shows that d = N in the absence of correlations, but d < 1/ρ<sup>2</sup> in the presence of correlations. These properties remain approximately true if the variances σ 2 i of the firing rates are drawn from a distribution with mean E - σ 2 i = σ 2 and variance Var- σ 2 i = δσ 4 . As Equation (9) shows, in such a case dimensionality is reduced compared to the case of equal variances, for example d ≈ σ 4 σ 4+δσ <sup>4</sup> N < N for large N when ρ = 0, δρ = 0.

The analytical results are shown in **Figure 7E** (full lines correspond to Equation 8), together with their estimates ("+") based on 1000 data points (same number as trials in **Figure 7D**; see Methods). The estimates are based on surrogate datasets

FIGURE 7 | Dimensionality and correlation. (A) Empirical single neuron firing rate distributions in the data (left) and in the model (right), for ongoing (black), and evoked activity (red). The distributions are approximately lognormal. X-axis, Firing rate (spks/s); Y-axis, frequency. (B) Example of independent Poisson spike trains with firing rates matched to the firing rates obtained in simulations of the spiking network model. (C) Example of correlated Poisson spike trains with firing rates matched to the firing rates obtained in simulations of the spiking network model. Pair-wise correlations of ρ = 0.1 were used (see Methods). X-axis, time [s]; Y-axis, neuron index. (D) Dimensionality as a function of ensemble size N in an ensemble of Poisson spike trains with spike count correlations ρ = 0, 0.1, 0.2 and firing rates matched to the model simulations of Figure 6. Dashed lines represent the fit of Equation (16) to the data (with δρ <sup>2</sup> = αρ <sup>2</sup>, σ <sup>4</sup> = δσ <sup>4</sup> = β), with best-fit parameters (mean ± s.e.m.) α = 0.22 ± 10−5, β = 340 ± 8. Filled circles (from top to bottom): dimensionality of the data (raster plots) shown in (B,C) (shaded areas represent SD). X-axis, ensemble size; Y-axis, dimensionality. (E) Theoretical prediction for the dependence of dimensionality on ensemble size N and firing rate correlation ρ for the case of uniform correlation, Equation (8) (thick lines; green to cyan to blue shades represent increasing correlations). "+" are dimensionality estimates from NT = 1, 000 trials for each N (same NT as in panel D, each trial providing a firing rate value sampled from a log-normal distribution), in the case of log-normally distributed firing rate variances σ 2 i with mean σ <sup>2</sup> = 40 (spk/s) <sup>∧</sup>2 and standard deviation 0.5 σ <sup>2</sup>. Theoretical predictions from Equation (16) match the estimated values in all cases (dashed black lines). X-axis, ensemble size N; Y-axis, dimensionality.

with lognormal-distributed variances σ 2 i to mimic the empirical distribution of variances found in GC (not shown).

# Estimation Bias

Comparison of **Figures 7D,E** shows that the dimensionality of the homogeneous network is underestimated compared to the theoretical value given by Equation (8). This is due to a finite number of trials and the presence of unequal variances with spread δσ 4 ("+" in **Figure 7E**). As **Figure 7E** shows, taking this into account will reduce the dimensionality to values comparable to those of the homogeneous network of **Figure 7D**. The dimensionality in that case is well predicted by Equation (16) (broken lines in **Figure 7E**). The same Equation (16) was fitted successfully to the data in **Figure 7D** (dashed) by tuning 2 parameters to account for the unknown variance and correlation width of the firing rates (see Methods for details).

Empirically, estimates of the dimensionality Equation (2) based on a finite number N<sup>T</sup> of trials tend to underestimate d (**Figure 3E**). The approximate estimator Equation (16) confirms that, for any ensemble size N, d is a monotonically increasing function of the number of trials (**Figure 8A**). Note that this holds for the mean value of the estimator (Equation 16) over many datasets, not for single estimates, which could overestimate the true d (not shown). Equation (16) also provides an excellent description of dimensionality as a function of firing rates' variance δσ 4 (**Figure 8B**) and pair-wise correlations width δρ 2 (**Figure 8C**). In particular, the mean and the variance of the pair-wise correlations have an interchangeable effect on d (see Equation 16); they both decrease the dimensionality and so does the firing rate variance δσ 4 (**Figure 8B**).

# Scaling of Dimensionality in the Presence of Clusters

We next compared the dimensionality of the homogeneous network's activity to that predicted by the clustered network model of **Figure 4**. To allow comparison with the homogeneous network, dimensionality was computed based on the spike counts in 200 ms bins rather than the HMM's firing rate vectors as in **Figure 6** (see Methods for details).

We found that the dependence of d on N in the clustered network depends on how the neurons are sampled. If the sampling is completely random, so that any neuron has the same probability of being added to the ensemble regardless of cluster membership, a concave dependence on N will appear, much like the case of the homogeneous network (**Figure 9A**, dashed lines). However, if neurons are selected one from each cluster until all clusters have been sampled once, then one neuron from each cluster until all clusters have been sampled twice, and so on, until all the neurons in the network have been sampled, then the dependence of d on N shows an abrupt transition when N = Q, i.e., when the number of sampled neurons reaches the number of clusters in the network (**Figure 9A**, full lines; see **Figure 9B** for raster plots with Q = 30 and N = 50). In the following, we refer to this sampling procedure as "ordered sampling," as a reminder that neurons are selected randomly from each cluster, but the clusters are selected in serial order. For N ≤ Q, the dimensionality grows linearly with ensemble size in both ongoing (slope 0.24 ± 0.01, r = 0.79, p < 10−10, black line) and evoked periods (slope 0.19 ± 0.01, r = 0.84, p < 10−10; red line), and was larger during ongoing than evoked activity [trial-matched two-way ANOVA, main effect:

FIGURE 8 | Dimensionality estimation. (A) Dependence of dimensionality on the number of trials for variable ensemble size N, for fixed correlations ρ = 0.1 and firing rates variances σ 2 i with mean σ <sup>2</sup> and standard deviation δσ <sup>2</sup> = 0.4 σ <sup>2</sup>. Dashed lines: theoretical prediction, Equation (16); dots: mean values from simulations of 20 surrogate datasets containing 10–1000 trials each (shaded areas: SD), with darker shades representing increasing number of trials. X-axis: ensemble size; Y-axis, dimensionality. (B) Dependence of dimensionality on the spread δσ <sup>2</sup> of the firing rates variances for fixed correlations ρ = 0.1 and firing rate variance with mean σ 2. Dashed lines: theoretical prediction, Equation (16); dots: mean values from simulations of 20 surrogate datasets containing 1000 trial each (shaded areas: SD), with lighter shades representing increasing values of δσ <sup>2</sup>/σ <sup>2</sup>). X-axis, ensemble size; Y-axis, dimensionality. (C) Dependence of dimensionality on the width δρ = p Var(ρ) of pair-wise firing rate correlations (with zero mean, ρ = 0), for firing rates variances σ 2 i with mean σ <sup>2</sup> and standard deviation δσ <sup>2</sup> = 0.4 σ <sup>2</sup>. Dashed lines: theoretical prediction, Equation (16); dots: mean values from simulations of 20 surrogate datasets containing 1000 trials each (shaded areas: SD), with darker shades representing increasing values of δρ. Inset: distribution of correlation coefficients used in the main figure. X-axis, ensemble size; Y-axis, dimensionality. In all panels, σ <sup>2</sup> = 40 (spk/s) <sup>∧</sup>2.

FIGURE 9 | Dimensionality in a clustered network. (A) Trial-matched dimensionality as a function of ensemble size in the recurrent network model (ongoing and evoked activity in black and red, respectively, with shaded areas representing s.e.m.). Filled lines represent ordered sampling, where ensembles to the left of the green vertical line (N = Q = 30) contain at most one neuron per cluster, while to the right of the line they contain one or more neurons from all clusters (filled circles indicate representative trials in panel B). Dashed lines represent random sampling of neurons, regardless of cluster membership. X-axis, ensemble size; Y-axis, dimensionality. (B) Representative trial of an ensemble of 50 neurons sampled from the recurrent network in Figure 4 during ongoing activity (upper plot, in black) or evoked activity (lower plot, in red) for the case of "ordered sampling" (full lines in panel A). Neurons are sorted according to their cluster membership (adjacent neuron pairs with similar activity belong to the same cluster, for neurons #1 up to #40; the last 10 neurons are sampled from the remaining clusters). X-axis, time to stimulus presentation at t = 0 (s); Y-axis, neuron index. (C) Average correlation matrix for 20 ensembles of N = 50 neurons from the clustered network model with Q = 30 clusters. For the first 40 neurons, adjacent pairs belong to the same cluster; the last 10 neurons (delimited by a dashed white square) belong to the remaining clusters (neurons are ordered as in panel B). Thus, neurons 1, 3, 5, …, 39 (20 neurons) belong to the first 20 clusters; neurons 2, 4, 6, …, 40 (20 neurons) belong also the first 20 clusters; and neurons 41, 42, 43, …, 50 (10 neurons) belong to the remaining 10 clusters. X-axis, Y-axis: neuron index. (D) Plot of Equation (12) giving d vs. N and ρ (uniform within-cluster correlations) for the sampling procedure of panel (B). X-axis, ensemble size N; Y-axis, dimensionality.

F(1, <sup>948</sup>) = 168, p < 10−30; interaction: F(5, <sup>948</sup>) = 4.1, p < 0.001].

These results are in keeping with the empirical and model results based on the HMM analysis (**Figures 3**, **6**). However, in the case of ordered sampling, the dependence of dimensionality on ensemble size tends to disappear for N ≥ Q both during ongoing (slope 0.010 ± 0.003, r = 0.1, p < 0.001) and evoked periods (slope 0.009 ± 0.002, r = 0.13, p < 10−<sup>4</sup> ; **Figure 9A**, full lines). The average dimensionality over the range 30 ≤ N ≤ 100 was significantly larger for ongoing, dongoing = 8.74 ± 0.06, than for evoked activity, devoked = 7.15±0.04 [trial-matched two-way ANOVA, main effect: F(1,2212) = 488, p < 10−30], confirming that dimensionality during ongoing is larger than during evoked activity also in this case. The difference in dimensionality between ongoing and evoked activity also holds in the case of random sampling on the entire range of N values (**Figure 9A**, dashed lines), confirming the generality of this finding.

# Dimensionality is Larger in the Presence of Clusters

Intuitively, the dimensionality saturates at N = Q in the clustered network because additional neurons will be highly correlated with already sampled ones. For N ≤ Q, each new neuron's activity adds an independent degree of freedom to the neural dynamics and thus increases its dimensionality. For Q > N, additional neurons are highly correlated with an existing neuron, adding little or no additional contribution to d. Indeed, compared to the low overall correlations found across all neuron pairs in the data (and used as desiderata for the homogeneous network), neurons belonging to the same model cluster had a much higher correlation of ρ = 0.92 [0.56, 0.96] (median and [25, 75] percentile), while neurons belonging to different clusters had negligible correlation (ρ ≈ 0, [−0.10, 0.06]). A negligible median correlation was typical: for example, negligible was the overall median correlation regardless of cluster membership (ρ ≈ 0 [−0.109, 0.083]); and the empirical correlation both during ongoing ([−0.047, 0.051], with rare maximal values of ρ ∼ 0.5), and evoked activity ([−0.085, 0.113], with rare maximal values of ρ ∼ 0.9). While we note the qualitative agreement of model and empirical correlations, we emphasize that these numbers were obtained using 200 ms bins and that they were quite sensitive to bin duration. In particular, the maximal correlations (regardless of sign) were substantially reduced for smaller bin durations (not shown).

Plugging these values into a correlation matrix reflecting the clustered architecture and the "ordered" sampling procedure used in **Figure 9B**, we obtained the matrix shown in **Figure 9C**, where pairwise correlations depend on whether or not the neurons belong to the same cluster (for the first 40 neurons, adjacent pairs belong to the same cluster; the last 10 neurons belong to the remaining clusters). It is natural to interpret such correlation matrix as the noisy observation of a blockdiagonal matrix such that neurons in the same cluster have uniform correlation while neurons from different clusters are uncorrelated. For such a correlation matrix the dimensionality can be evaluated exactly (see Equation 12 of Methods). In the approximation where all neurons have the same variance, this reduces to Equation (13), i.e.,

$$d(N,\rho) = \begin{cases} \begin{array}{c} N \\ \hline \frac{N}{1+m\rho^2\left[1-\left(Q-p\right)/N\right]} \end{array} , \quad N > Q \end{cases},$$

where N = mQ + p. This formula is plotted in **Figure 9D** for relevant values of ρ and N and it explains the origin of the abrupt transition in dimensionality at Q = N. (The reasons for a dimensionality lower than N for N ≤ Q in the data–see **Figure 9A**–are, also in this case, the finite number of data points (250) used for its estimation and the non-uniform distributions of firing rate variances and correlations).

Note that the formula also predicts cusps in dimensionality (which become local maxima for large ρ) whenever the ensemble size is an exact multiple of the number of clusters. This is also visible in the simulated data of **Figure 9A**, where local maxima seem to appear at N = 30, 60, 90 with Q = 30 clusters. It is also worth mentioning that, for low intra-cluster correlations, the dependence on N predicted by Equation (13) becomes smoother and the cusps harder to detect (not shown), suggesting that the behavior of a clustered network with weak clusters tends to converge to the behavior of a homogeneous asynchronous network—therefore lacking sequences of hidden states. Thus, the complexity of the network dynamics is reflected in how its dimensionality scales with N, assuming that one may sample one neuron per cluster (i.e., via "ordered sampling").

Even though it is not clear how to perform ordered sampling empirically (see Discussion), this result is nevertheless useful since it represents an upper bound also in the case of random sampling (see **Figure 9A**, dashed lines). Equation (13) predicts that d ≤ Q/ρ<sup>2</sup> , with this value reached asymptotically for large N. In the case of random sampling, growth to this bound is even slower (**Figure 9A**). For comparison, in a homogeneous network d ≤ 1/ρ<sup>2</sup> from Equation (8), a bound that is smaller by a factor of Q. Finally, homogeneous dimensionality is dominated by clustered dimensionality also in the more realistic case of nonuniform variances and correlations, where similar bounds are found in both cases (see Methods for details).

# DISCUSSION

In this paper we have investigated the dimensionality of the neural activity in the gustatory cortex of alert rats. Dimensionality was defined as a collective property of ensembles of simultaneously recorded neurons that reflects the effective space occupied by the ensemble activity during either ongoing or evoked activity. If one represents ensemble activity in terms of firing rate vectors, whose dimension is the number of ensemble neurons N, then the collection of rate vectors across trials takes the form of a set of points in the N-dimensional space of firing rates. Roughly, dimensionality is the minimal number of dimensions necessary to provide an accurate description of such set of points, which may be localized on a lower-dimensional subspace inside the whole firing rate space.

One of the main results of this paper is that the dimensionality of evoked activity is smaller than that of ongoing activity, i.e., stimulus presentation quenches dimensionality. More specifically, the dimensionality is linearly related to the ensemble size, with a significantly larger slope during ongoing activity compared to evoked activity (compare **Figures 3B,D**). We explained this phenomenon using a biologically plausible, mechanistic spiking network model based on recurrent connectivity with clustered architecture. The model was recently introduced in Mazzucato et al. (2015) to account for the observed dynamics of ensembles of GC neurons as sequences of metastable states, where each state is defined as a vector of firing rates across simultaneously recorded neurons. The model captures the reduction in trial-to-trial variability and the multiple firing rates attained by single neurons across different states observed in GC upon stimulus presentation. Here, the same model was found to capture also the stimulus-induced reduction of dimensionality. While the set of active clusters during ongoing activity varies randomly, allowing the ensemble dynamics to explore a large portion of firing rate space, the evoked set of active clusters is limited mostly to the stimulusselective clusters only (see Mazzucato et al., 2015 for a detailed analysis). The dynamics of cluster activation in the model thus explains the more pronounced dependence of dimensionality on ensemble size found during ongoing compared to evoked activity.

We presented a simple theory of how dimensionality depends on the number of simultaneously recorded neurons N, their firing rate correlations, their variance, and the number and duration of recording trials. We found that dimensionality increases with N and decreases with the amount of pair-wise correlations among the neurons (e.g., **Figure 8C**). At parity of correlations, dimensionality is maximal when all neurons have the same firing rate variance, and it decreases as the distribution of count variances becomes more heterogeneous (e.g., **Figure 8B**). The estimation of dimensionality based on a finite dataset is an increasing function of the number of trials (**Figure 8A**). Finally, introducing clustered correlations in the theory, and sampling one neuron per cluster as in **Figure 9B**, results in cusps at values of N that are multiples of the number of clusters (**Figure 9D**), in agreement with the predictions of the spiking network model (**Figure 9A**, full lines).

# Dimensionality Scaling with Ensemble Size

The increased dimensionality with sample size, especially during ongoing activity, was found empirically in datasets with 3–9 neurons per ensemble, but could be extrapolated for larger N in a spiking network model with homogeneous or clustered architecture. In homogeneous networks with finite correlations the dimensionality is predicted to increase sub-linearly with N (Equation 8), whereas in the clustered network it may exhibit cusps at multiple values of the number of clusters (**Figure 9A**), and would saturate quickly to a value that depends on the ratio of the number of clusters Q and the amount of pair-wise correlations, d ≤ Q/ρ<sup>2</sup> . Testing this prediction requires the ability to sample neurons one from each cluster, until all clusters are sampled, and seems beyond the current recording techniques. However, looking for natural groupings of neurons based on response similarities could uncover spatial segregation of clusters (Kiani et al., 2015) and could perhaps allow sampling neurons according to this procedure. Moreover, the model predicts a slower approach to a similar bound also in the case of random sampling.

Dimensionality in a homogeneous network is instead bounded by 1/ρ<sup>2</sup> , and hence it is a factor Q smaller than in the clustered network. Dimensionality is maximal in a population of independent neurons (ρ = 0), where it grows linearly with N; however, neurons of recurrent networks have wide-ranging correlations (see e.g., **Figures 6E,F** and its empirical counterpart, **Figures 3G,H**). Since the presence of even low correlations can dramatically reduce the dimensionality (see **Figure 7D**), the neural activity in a clustered architecture can reach much higher values at parity of correlations, representing an intermediate case between a homogeneous network and a population of independent neurons.

Evidence for the presence of spatial clusters has been recently reported in the prefrontal cortex based on correlations analyses (Kiani et al., 2015). An alternative possibility is that neural clusters are not spatially but functionally arranged, and cluster memberships vary with time and task complexity (Rickert et al., 2009). Can our model provide indirect tools to help uncover the presence of clusters? A closer look at **Figures 6E,F** reveal a small peak at large correlations due to the contribution of highly correlated neurons belonging to the same cluster. This peak would be absent in a homogenous network and thus is the signature of a clustered architecture. However, such peak is populated by only small fraction (1/Q) of the total number of neuron pairs, which hinders its empirical detection (no peak at large correlations is clearly visible in our data, see **Figures 3G,H**).

# Dimensionality and Trial-to-Trial Variability

Cortical recordings from alert animals show that neurons produce irregular spike trains with variable spike counts across trials (Shadlen and Newsome, 1994; Fontanini and Katz, 2008; Moreno-Bote, 2014). Despite many efforts, it remains a key issue to establish whether variability is detrimental (Gur et al., 1997; White et al., 2012) or useful (McDonnell and Ward, 2011) for neural computation.

Trial-to-trial variability is reduced during preparatory activity (Churchland et al., 2006), during the presentation of a stimulus (Churchland, 2010b), or when stimuli are expected (Samuelsen et al., 2012), a phenomenon that would not occur in a population of independent or homogeneously connected neurons (Litwin-Kumar and Doiron, 2012). Recent work has shown that the stimulus-induced reduction of trial-to-trial variability can be due to spike-frequency adaptation in balanced networks (Farkhooi et al., 2013) or to slow dynamic fluctuations generated in a recurrent spiking networks with clustered connectivity (Deco and Hugues, 2012; Litwin-Kumar and Doiron, 2012; Mazzucato et al., 2015). In clustered network models, slow fluctuations in firing rates across neurons can ignite metastable sequences of neural activity, closely resembling metastable sequences observed experimentally (Abeles et al., 1995; Seidemann et al., 1996; Jones et al., 2007; Kemere et al., 2008; Durstewitz et al., 2010; Ponce-Alvarez et al., 2012; Mazzucato et al., 2015). The slow, metastable dynamics of cluster activation produces high variability in the spike count during ongoing activity. While cluster activations occur at random times during ongoing activity periods, stimulus presentation locks cluster activation at its onset, leading to a decrease in trial-to-trial variability.

Similarly, a stimulus-induced reduction of dimensionality is obtained in the same model. In this case, preferred cluster activation due to stimulus onset generates an increase in pairwise correlations that reduce dimensionality. Note that the two properties (trial-to-trial variability and dimensionality) are conceptually distinct. An ensemble of Poisson spike trains can be highly correlated (hence have low dimensionality), yet the Fano Factor of each spike train will still be 1 (hence high), independently of the correlations among neurons. In a recurrent network, however, dimensionality and trial-to-trial variability may become intertwined and exhibit similar properties, such as the stimulus-induced reduction observed in a model with clustered connectivity. A deeper investigation of the link between dimensionality and trial-to-trial variability in recurrent networks is left for future studies.

# Alternative Definitions of Dimensionality

Following (Abbott et al., 2011) we have defined dimensionality (Equation 2) as the dimension of an effective linear subspace of firing rate vectors containing the most variance of the neural activity. It differs somewhat from the typical dimensionality reduction based on PCA that retains only the number of eigenvectors explaining a predefined amount of variance (see Broome et al., 2006; Geffen et al., 2009), because Equation (2) includes contribution from all eigenvalues. Moreover, we have computed the firing rate correlations in bins of variable width that match the duration of the HMM states. Although, our main results do no depend on bin size (see Results' section "Time course of dimensionality as a function of ensemble size"), the actual value of dimensionality decreases with increasing bin duration. Thus, any choice of bin size (e.g., 200 ms in **Figures 3F,G**) remains somewhat arbitrary. A better method is to use a variable bin size as dictated by the HMM analysis, as done in **Figures 3B–D**. This method also prevents diluting correlations among firing rates that would occur if one neuron were to change state inside the current bin, because during a hidden state the firing rates of the neurons are constant (by definition). Thus, this provides a principled adaptive procedure for selecting the bin size and eliminates the dependence of dimensionality on the bin width used for the analysis.

Other definitions of neural dimensionality have been proposed in the literature, which aim at capturing different properties of the neural activity, typically during stimulus-evoked activity. A measure of dimensionality related to ours, and referred to as "complexity," was introduced in Cadieu et al. (2013). According to their definition, population firing rate vectors from all evoked conditions were first decomposed along their kernel Principal Components (Montavon et al., 2011). A linear classifier was then trained on an increasing number of leading PCs in order to perform a discrimination task, where the number of PCs used was defined as the complexity of the representation. In general, the classification accuracy improves with increasing complexity, and it may saturate when all PCs containing relevant features are used—with the remaining PCs representing noise or information irrelevant to the task. Reaching high accuracy at low complexity implies good generalization performance, i.e., the ability to classify novel variations of a stimulus in the correct category. Neural representations in monkey inferotemporal cortex (IT) were found to require lower complexity than in area V4, confirming IT's premier role in classifying visual objects despite large variations in shape, orientation and background (Cadieu et al., 2013). Complexity relies on a supervised algorithm and is an efficient tool to capture the generalization properties of evoked representations (see DiCarlo et al., 2012) for its relevance to visual object recognition).

A second definition of dimensionality, sometimes referred to as "shattering dimensionality" in the Machine Learning literature, has been used to assess the discrimination properties of the neural representation (Rigotti et al., 2013). Given a set of p firing rate vectors, one can split them into two classes (e.g., white and black colorings) in 2<sup>p</sup> different ways, and train a classifier to learn as many of those binary classification labels as possible. The shattering dimensionality is then defined as (the logarithm of) the largest number of binary classifications that can be implemented. This measure of dimensionality was found to drop significantly in monkey prefrontal cortex during the error trials of a recall task, and thus predicts the ability of the monkey to correctly perform the task (Rigotti et al., 2013).

A flexible and informative neural representation is one that achieves a large shattering dimensionality (good discrimination) while keeping a low complexity (good generalization). Note that both complexity and shattering dimensionality represent measures of classification performance in task-related paradigms, and their definition requires a set of evoked conditions to be classified via a supervised learning algorithm. While both definitions could be applied to neural activity in our stimulusevoked data, their interpretation is not readily extended to periods of ongoing activity, as the latter is not associated to desired targets in a way that can be learned by a classification algorithm. Since our main aim was to compare the dimensionality of ongoing and evoked activity, the unsupervised approach of Abbott et al. (2011) and their notion of "effective" dimensionality was better suited for our analysis. A related definition of dimensionality has been used by Gao and Ganguli (2015) to investigate neural representations of movements in motor cortex.

Many measures of dimensionality used in the literature (including ours and some of those discussed above) are based on pair-wise correlations. However, neural activity is known to give rise also to higher-order correlations (Martignon et al., 2000). Given that the extent and relevance of higher-order correlations is actively debated (Schneidman et al., 2006; Staude et al., 2010), it would be useful to include them in measures of dimensionality. This is left for a future study.

# Ongoing Activity and Task Complexity

The relationship between ongoing and stimulus-evoked activity has been linked to the functional connectivity of local cortical circuits, and their mutual relationship has been the object of both theoretical and experimental investigations, often with contrasting conclusions (e.g., Arieli et al., 1996; Tsodyks et al., 1999; Kenet et al., 2003; Luczak et al., 2009; Tkacik et al., 2010; Berkes et al., 2011; Mazzucato et al., 2015). Here, we have focused on the dimensionality of ongoing and evoked activity and have shown that neural activity during ongoing periods occupies a space of larger dimensionality compared to evoked activity. Although, based on a different measure of dimensionality, recent results on the relation between the dimensionality of evoked activity and task complexity suggest that evoked dimensionality is roughly equal to the number of task conditions (Rigotti et al., 2013). It is natural to ask whether the dimensionality of ongoing activity provides an estimate of the complexity of the hardest task that can be supported by the neural activity. Moreover, based on the clustered network model, the presence of clusters imposes an upper value d ≤ Q/ρ<sup>2</sup> during ongoing activity, suggesting that a discrimination task with up to ∝ Q different conditions may be supported. The experience of taste consumption is by itself multidimensional, including chemo- and oro-sensory aspects (i.e., taste identity Jezzini et al., 2013, and concentration Sadacca et al., 2012, texture, temperature, Yamamoto et al., 1981, 1988) as well as psychological aspects (hedonic value Katz et al., 2001; Grossman et al., 2008, anticipation Samuelsen et al., 2012; Gardner and Fontanini, 2014, novelty Inberg et al., 2013; Bermudez-Rattoni, 2014, and satiety effects de Araujo et al., 2006). It is tempting to speculate that neural activity during ongoing periods explores all these different dimensions, while evoked activity is confined to the features of the particular taste being delivered or attended in a specific context.

# REFERENCES


Establishing a precise experimental and theoretical link between the number of clusters and task complexity is an important question left for future studies.

# ACKNOWLEDGMENTS

This work was supported by a National Institute of Deafness and Other Communication Disorders Grant K25-DC013557 (LM), by the Swartz Foundation Award 66438 (LM), by National Institute of Deafness and Other Communication Disorders Grant R01-DC010389 (AF), by a Klingenstein Foundation Fellowship (AF), and by a National Science Foundation Grant IIS-1161852 (GL). We thank Drs. Stefano Fusi and Memming Park for useful discussions and David Ecker at the Research Technologies DoIT of Stony Brook University for access to its computational resources.

widespread cortical phenomenon. Nat. Neurosci. 13, 369–378. doi: 10.1038/nn. 2501


spiking cortical neurons. J. Neurophysiol. 96, 3448–3464. doi: 10.1152/jn.004 53.2006


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Mazzucato, Fontanini and La Camera. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Statistical Frequency-Dependent Analysis of Trial-to-Trial Variability in Single Time Series by Recurrence Plots

Tamara Tošic´ 1, 2, 3 \*, Kristin K. Sellers 4, 5, Flavio Fröhlich4, 5, 6, 7, 8, Mariia Fedotenkova1, 2, 3 , Peter beim Graben9, 10 and Axel Hutt 1, 2, 3

<sup>1</sup> Team Neurosys, Inria, Villers-lès-Nancy, France, <sup>2</sup> Loria, Centre National de la Recherche Scientifique, UMR no 7503, Villers-lès-Nancy, France, <sup>3</sup> Université de Lorraine, Loria, UMR no 7503, Villers-lès-Nancy, France, <sup>4</sup> Department of Psychiatry, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, <sup>5</sup> Neurobiology Curriculum, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, <sup>6</sup> Department of Cell Biology and Physiology, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, <sup>7</sup> Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, <sup>8</sup> Neuroscience Center, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA, <sup>9</sup> Department of German Studies and Linguistics, Berlin, Germany, <sup>10</sup> Bernstein Center for Computational Neuroscience, Berlin, Germany

#### Edited by:

Emili Balaguer-Ballester, Bournemouth University, UK and Bernstein Center for Computational Neuroscience Heidelberg-Mannheim, Germany

#### Reviewed by:

Jorge Otero-Millan, Johns Hopkins University, USA Hazem Toutounji, Central Institute of Mental Health, Germany

> \*Correspondence: Tamara Tošic´ tamtos@gmail.com

Received: 31 March 2015 Accepted: 18 December 2015 Published: 14 January 2016

#### Citation:

Tošic T, Sellers KK, Fröhlich F, ´ Fedotenkova M, beim Graben P and Hutt A (2016) Statistical Frequency-Dependent Analysis of Trial-to-Trial Variability in Single Time Series by Recurrence Plots. Front. Syst. Neurosci. 9:184. doi: 10.3389/fnsys.2015.00184 For decades, research in neuroscience has supported the hypothesis that brain dynamics exhibits recurrent metastable states connected by transients, which together encode fundamental neural information processing. To understand the system's dynamics it is important to detect such recurrence domains, but it is challenging to extract them from experimental neuroscience datasets due to the large trial-to-trial variability. The proposed methodology extracts recurrent metastable states in univariate time series by transforming datasets into their time-frequency representations and computing recurrence plots based on instantaneous spectral power values in various frequency bands. Additionally, a new statistical inference analysis compares different trial recurrence plots with corresponding surrogates to obtain statistically significant recurrent structures. This combination of methods is validated by applying it to two artificial datasets. In a final study of visually-evoked Local Field Potentials in partially anesthetized ferrets, the methodology is able to reveal recurrence structures of neural responses with trial-to-trial variability. Focusing on different frequency bands, the δ-band activity is much less recurrent than α-band activity. Moreover, α-activity is susceptible to pre-stimuli, while δ-activity is much less sensitive to pre-stimuli. This difference in recurrence structures in different frequency bands indicates diverse underlying information processing steps in the brain.

Keywords: trial-to-trial variability, time-frequency analysis, local field potentials, recurrence plot analysis, statistical inference, surrogate data, anesthesia, ferret

# 1. INTRODUCTION

Investigation of metastable states (MS) and transients of complex dynamical systems has become increasingly important over the last decades. In this context, dynamical systems spend longer time intervals in MSs than in transients between MSs. The large interest in studying such states comes from the belief that a complex temporal behavior of systems may be decomposed into a simple sequence of alternating MSs and transients between them. This reduced description is a model that captures the essential dynamic elements of rather complex underlying dynamics. Applications range from spin glasses (Larralde and Leyvraz, 2005) to molecular configurations (Deuflhard and Weber, 2005) and geoscientific applications (Froyland et al., 2007). In neuroscience, the related concept of sequential metastable attractors has received increasing attention in the last years (Friston, 1997; Oullier and Kelso, 2006; Rabinovich et al., 2008b; Yildiz and Kiebel, 2011; Hudson et al., 2014; Tognoli and Kelso, 2014). Primarily, works are motivated by the experimental observation of signal features showing alternations of dynamical behavior at fast and slow time scales (Hutt and Riedel, 2003; Hutt, 2004; Mazor and Laurent, 2005; Allefeld et al., 2009).

Originally the concept of metastability refers to slow relaxation dynamics in statistical physics (Larralde and Leyvraz, 2005; Tokman et al., 2011). In a much wider sense, this notion is nowadays used for regions in the phase space of a dynamical system with relatively large dwell that are connected by transients (Friston, 1997; Rabinovich et al., 2008b; Tognoli and Kelso, 2014). Paradigmatic examples for those MSs are almost invariant sets (Froyland, 2005) and recurrence domains (beim Graben and Hutt, 2013), such as saddles connected by heteroclinic trajectories (Rabinovich et al., 2008a) or the "wings" of the Lorenz attractor (Lorenz, 1963). For this attractor in particular, it is attractive itself and has two recurrence domains centered around two unstable foci. Geometrically, these domains are spatially separated and the system's trajectory alternately approaches to and departs from the foci. The system spends much longer time in the vicinity of a focus compared to transient intervals between the two foci. Therefore, one may refer to a Lorenz wing as to a MS: the system remains for a longer time in one partition cell of the phase space before it performs a rapid transition to another partition cell of the phase space. A MS is thus identified with a recurrence domain, while non-recurrent portions of a trajectory can be compared with transients.

In neuroscience, metastability assumed increasing experimental evidence over recent years. Lehmann et al. (1987), Wackermann et al. (1993) observed sequences of metastable electroencephalogram (EEG) topographies, which they called brain microstates. Hutt and Riedel (2003), Hutt (2004), beim Graben and Hutt (2015) argued that components of the event-related brain potentials (ERPs) reflecting perceptional and cognitive processes could be identified with metastable brain states. Mazor and Laurent, for instance, reported sequences of metastable states in a reconstructed activation space of the locust's neural odor circuit (Mazor and Laurent, 2005). Allefeld et al. (2009) were able to detect metastable states in epileptic EEG time series through spectral clustering methods, and most recently, Hudson et al. (2014) revealed metastable transition networks in the recovery from anesthesia. Consequently, to understand underlying neural mechanisms much better, it is necessary to develop advanced techniques to detect these recurrence structures in experimental time series.

For the identification of metastability in time series, their characteristic slow time scales must be separated from the fast dynamics of phase space trajectories. The method known as Perron clustering (Deuflhard and Weber, 2005), separates the system's phase states into partitions that can approximate Markov chain states (Deuflhard and Weber, 2005; Froyland, 2005; Larralde and Leyvraz, 2005; Gaveau and Schulman, 2006; Allefeld et al., 2009). Applying spectral clustering methods to the resulting transition matrix yields the time scales of the process, while their corresponding (left-)eigenvectors allow the unification of cells into a partition of metastable states (Gaveau and Schulman, 2006; Allefeld et al., 2009). Another approach by Hutt and Riedel (2003) utilizes the slowing-down of the system's trajectory in the vicinity of saddles by means of phase space clustering. Most recently, beim Graben and Hutt suggested to combine recurrence plot techniques and symbolic dynamics in order to partition a system's phase space into its recurrence domains (beim Graben and Hutt, 2013, 2015). The application of the latter method to experimental event-related potentials has identified metastable attractors to socalled ERP-components, known to reflect cognitive processing stages in neural information processing.

Developing novel analysis tools for representation and tracking of non-linear transient patterns faces numerous challenges, such as reducing the signal dimensionality while preserving the information significant for the detection task or building methods robust to acquisition noise. Recurrence analysis has been used for identifying transient patterns in experimental EEG (Shalbaf et al., 2015), for classifying patients based on EEG time series (McCarthy et al., 2014) and for prediction of responses during anesthesia (Huang et al., 2006). A key feature of recurrence analysis is to identify sequential states in a multi-dimensional signal space, as shown in most previous studies (beim Graben and Hutt, 2013, 2015). If the experimental data under study is multi-dimensional, for instance a multichannel EEG recording, the data serves directly as the input to the recurrence analysis. However, it is not valid to compute recurrence plots in the case of univariate time series and hence the data can not be analyzed directly. Therefore, it is necessary to transform the univariate signal to a multivariate (multidimensional) signal. Typically this is done by delay-embedding techniques (Webber and Zbilut, 1994; Iwanski and Bradley, 1998) inspired by Takens' theorem (Takens, 1981). The corresponding embedding dimension and delay time in these techniques are chosen rather independent from the dynamic features of the data since typically these are not known a priori.

In neuroscience, patterns occurring in certain frequency bands play distinct roles in neural information processing (Kandel et al., 2000; Schnitzler and Gross, 2005). We argue that this additional knowledge can be taken into account and the present work proposes a novel technique based on timefrequency representations of univariate signals. Here, the signal is transformed into its time-frequency representation of spectral power which spans a new phase space in which the signal trajectory evolves. Hence, one may call this transformation spectral power embedding since the new phase space encodes instantaneous power in certain frequency bands. The additional advantage of this approach is that it permits to analyse the recurrence structure of data in selected frequency bands. For completeness, we mention that a signal is fully defined by its instantaneous amplitude and phase.

In this work we propose a new method for the detection of metastable states in univariate neural signals. To obtain statistically significant evidence of recurrence structures in signals, we conduct a statistical test over the set of novel, frequency-selective recurrence plots (RP). Below we describe methodologies for building such frequency-selective RPs and performing statistical inference tests. These tests indicate how stable the recurrence plots are with respect to trial-to-trial variability. This novel statistical evaluation is necessary in the analysis of neurophysiological data, since trial-to-trial variability is a well-known experimental finding in such signals. In our work we analyse synthetic transient oscillations and one state variable of the Lorenz attractor involving acquisition noise to validate the methodology. Finally, the study of experimental Local Field Potentials obtained in partially anesthetized ferrets (Mustela putorius furo) during a visual stimulus experiment allows to extract new insights into neural information processing. For instance, we show that temporal recurrence occurs in the α-frequency band but not in the δ-frequency band. This result suggests that in the α-band the brain processes the information step-wise (state by state) while no step-wise process is performed in the δ-band.

# 2. MATERIALS AND METHODS

In this section we introduce the novel method for studying temporal recurrences common in recurrence plots of different trials. Section 2.1 introduces classical recurrence plots and describes corresponding parameters. Then, we provide a novel method to compute recurrence plots from their time-frequency representations. In Section 2.2 we propose the statistical test method that analyses the similarity of RPs and finds their statistically significant parts. Finally, in Section 2.3 we describe the datasets used in this work.

# 2.1. Recurrence Plots and Novel Time-Frequency Representations

Recurrence is a fundamental property of dynamical systems which characterizes the behavior of the system in phase space (Poincaré, 1890). A recurrent signal instance is a moment in time when the trajectory returns to a neigborhood of a location in phase space it has already visited previously.

Deterministic dynamical systems are described by their trajectory. A trajectory x(t) ∈ R n , t ∈ R is sampled at times t = i1t, i ∈ {1, 2,. .. , N}, where 1t is the sampling time interval and N is the total number of samples. For notation simplicity, in this paper we denote the signal sample x(i1t) by x(i). Then, a recurrence plot (RP) is defined as the N × N matrix **R**, whose elements ri,<sup>j</sup> take values ri,<sup>j</sup> = 1 when two trajectory samples lay within the open ball B(ǫ) of radius ǫ

$$r\_{i,j} = \begin{cases} 1, & \text{if } d(\underline{\mathbf{x}}(i), \underline{\mathbf{x}}(j)) < \epsilon, \\ 0, & \text{otherwise}, \end{cases} \tag{1}$$

where d(·, ·) is a distance function and i, j ∈ {1,... , N}. Hence, recurrence plots are two-dimensional binary matrices obtained by distance based thresholding and its elements take values ri,<sup>j</sup> ∈ {0, 1}. In this work, pixels in RPs will be color-coded white for values ri,<sup>j</sup> = 0 and black otherwise.

For an arbitrary chosen ǫ value we can not guarantee that some of the significant dynamic features are not discarded by thresholding. To minimize such a thresholding error, we compute the optimal threshold value ǫ <sup>∗</sup> which maximizes the symbolic entropy for a given distance function, as proposed in beim Graben and Hutt (2013). In more detail, under the assumption that recurrence domains are uniformly distributed for a given recurrence plot, the method constructs disjunct and transitive symbolic recurrence plot matrices from multivariate data. This method permits to identify MSs in a recurrence plot and maps each state (and the transients between the states) to a symbol. Consequently, one maps the high-dimensional dynamics of the system to a sequence of symbols. Let p<sup>k</sup> be the probability of the occurrence of the state k, i.e., the number of the occurrences of the symbol k divided by the total number of occurrences of all symbols. Then maximizing the entropy

$$H(\epsilon) = -\frac{1}{\mathcal{S}\_{k,\epsilon}} \sum\_{k=1}^{\mathcal{S}\_{k,\epsilon}} p\_k \log(p\_k),\tag{2}$$

for a range of ǫ-values yields that value of ǫ for which the distribution of occurrence probabilities {p<sup>k</sup> } approaches uniformity, i.e., for which all states are equally probable. Here, Sk,ǫ is the number of states for a given ǫ. Then the optimal value

$$
\epsilon^\* = \arg\max\_{\epsilon} H(\epsilon).
$$

maximizes the entropy of the extracted symbolic sequence and hence the recurrence structure of the data. This optimal value is computed for each dataset separately.

After defining conventional RPs and computation of the optimal parameter ǫ, the remaining part of this section focusses on how to build frequency-selective recurrence plots. Many biophysiological signals have characteristic frequency signatures. For example, the human heart beats about sixty times per minute in average, i.e., at the frequency of 1 Hz. Another example are eye blinks that induce signal changes in the α-frequency band (frequencies in the interval 8–12 Hz) in EEG recordings. To take into account the distinct signatures of spectral bands present in neural signals, we propose a novel concept for building recurrence plots from time-frequency signal representations, instead of building them directly from univariate data or constructing them by employing delay-embedding techniques. Such representations, in general adapted for non-stationary signal analysis, give insights into frequency bands of importance and provide additional flexibility to recurrence plot analysis that is not present in time-domain, for e.g., the possibility to weight the importance of some frequency bands. In the literature there are several ways to choose values of the frequency bands. We use the following frequency interval definitions: the δ-frequency band denotes the interval [0.5 Hz; 4 Hz], the θ-frequency band the interval [4 Hz; 8 Hz], α-band [8 Hz; 12 Hz], β-band [12 Hz; 20 Hz], and the γ -band denotes the interval [20 Hz; 40 Hz].

We build novel recurrence plots in three steps, as shown in **Figure 1**: (i) we expand the set of T univariate trials

{**x**1, **x**2,... , **x**T} to their corresponding time-frequency domains; (ii) we compute the mean power of the spectrum over certain sets of frequencies. These mean power time series may be called **y**j(t) ∈ R S , j = 1,... , T, where S is the number of frequency bands; (iii) we compute recurrence plots by computing distances between vectors y(i) and y(j), i, j ∈ {1,... , N} as in Equation (1). In the following paragraphs we describe these blocks in more detail.

The first block in **Figure 1** provides a time-frequency representation of the signal. In classical spectrogram calculations, the stronger (weaker) is the localization of signals in time, the larger (smaller) are their localization windows in frequency. This effect is called the uncertainty principle implied in the Fourier transform. A synchrosqueezing (SSQ) transform overcomes this deficiency by performing wavelet-based filtering and signal power reassignment to the appropriate frequencies. In addition, Meignen et al. (2012) and Auger et al. (2013) show that SSQ is superior for processing neural signals when compared to conventional spectral analysis methods, such as continuous wavelet transform or spectrogram. Hence, we use the SSQ transform defined in Section 2.1.1 as the processing block (i) in **Figure 1**.

The second block in **Figure 1** computes the mean value of the power spectrum (MPS) for sets of frequencies, see Section 2.1.2 for details. This is one of the basic features for studying neural signals. We assume that the dynamics of the neural system encoded in frequencies is proportional to the power spectrum in sets of frequencies. This analysis step provides multi-variate time series whose dimension is equal to the resulting vector of averaged frequency bands.

Finally, in the third processing block in the figure we compute recurrence plots from the obtained time-frequency dataset as in Equation (1). If we do not explicitly mention otherwise, we use features from all the frequency bands to compute recurrence plots. In the experimental ferret dataset, we additionally present cases when recurrence plots are calculated from the single frequency band features such as δ- or α-frequency bands, since these bands play an important role in the loss of consciousness under anesthesia.

To summarize, the proposed method for building recurrence plots from time-frequency representations grasps band-related features and allows flexibility in the analysis of particular frequency bands, which is not possible in the classical RP analysis. Our approach however requires additional computations of the synchrosqueezing transform and mean power of the spectrum.

### 2.1.1. Synchrosqueezing Transform

For completeness of this work, in this section we provide the mathematical definition of the synchrosqueezing transform (Meignen et al., 2012; Auger et al., 2013), that we use as a processing block in the proposed algorithm, see **Figure 1**. We presume that input signals are composed of several components with time-varying oscillatory characteristics. In other words, we assume that signals f(t) can be well approximated with K signal components, f(t) = P<sup>K</sup> k=1 fk (t) + e(t), f<sup>k</sup> (t) = A<sup>k</sup> (t)e 2πiφ<sup>k</sup> (t) , where A<sup>k</sup> (t) and φ ′ k (t) = 1 2π dφ<sup>k</sup> (t) dt denote the amplitude and the instantaneous frequency (IF) of each component and e(t) represents a small error. We assume that the components f<sup>k</sup> have slowly time-varying amplitudes A<sup>k</sup> (t) and sufficiently smooth IFs. These conditions assure that signal components are well separated in frequencies and the complete definition is available in Thakur et al. (2013), Def. II.1 (codes available online in Thakur, 2013).

Let a wavelet ψ(t) be a square integrable and normalized function. Then, its scaled and time-shifted variants ψ( t−b a ) represent a set of scaled bandpass filters. In the following, we denote the frequency of one signal component by ω<sup>k</sup> ≈ 2π dφ<sup>k</sup> (t) dt . A Continuous wavelet transform (CWT) of the function f at scale a and time shift b is defined by W<sup>f</sup> (a, b) = √ 1 a R <sup>∞</sup> −∞ f(t)ψ( t−b a )dt, which represents a convolution of scaled and band-passed filters with the signal. The shifts of wavelet function are driven by the scale value a. For example, for the first signal component with frequency ω1, the value of the wavelet coefficient W<sup>f</sup> (a1, b) spreads around the scale factor a<sup>1</sup> = ωψ ω1 , where ω<sup>ψ</sup> is the central wavelet frequency. Therefore, the estimated IF in the neighborhood of this value of the scale is equal to the frequency ω1. The synchrosqueezing transform T(ωq, b) uses estimates of the instantaneous frequency ωf (a, b), computed for each scale-time pair (a, b) by ω<sup>f</sup> (a, b) = −iW<sup>f</sup> (a, b) ∂W<sup>f</sup> (a,b) ∂b to reallocate the energy of the wavelet coefficients. Let 1a<sup>p</sup> (1ω) denote resolution steps in scale P (frequency). Then, this transform, defined by T(ωq, b) = ap:|ω<sup>f</sup> (ap,b)−ωq|≤1ω/<sup>2</sup> W<sup>f</sup> (ap, b)a <sup>−</sup>3/21a<sup>p</sup> enhances frequency localization of oscillating components of the signal and provides more precise time-frequency representations of the signal. In analogy to the spectrogram used in classical short-time Fourier analysis, we plot values

$$S(a\_q, b) = |T(a\_q, b)|^2 \tag{3}$$

for each pair (ωq, b) in time-frequency plots, see **Figures 5A,E**, **6A,C**.

#### 2.1.2. Mean power spectrum

For each frequency band with Q components, the mean power spectrum value is defined by

$$MPS(t) = \frac{1}{Q} \sum\_{q=1}^{Q} S(\omega\_q, t),\tag{4}$$

where S(ωq, t) is defined in Equation (3), ω<sup>q</sup> are frequencies of one frequency band and t is time.

### 2.2. Statistical analysis

We study statistical properties of frequency-selective RPs obtained from time-frequency trial representations. By virtue of noise effects and an expected trial-to-trial variability, recurrence plot structures are expected to vary from trial to trial. To evaluate the recurrence plots statistically, we perform a statistical inference analysis based on a classical chi-squared test (Yates, 1934). To this end, we construct surrogate recurrence plots and employ an inference test.

Classically, surrogate sets of univariate signals (Schreiber and Schmitz, 2000) preserve some of the important features of the original time series, for example the spectrum magnitude, while they replace the phase values by a random sequence of values. The reasoning behind this randomization is that time domain reshaping destroys non-stationarities, so the local spectral components will vary while the global spectrum remains the same. As a consequence, the mean and variance of the signal do not change (Borgnat et al., 2010; Richard et al., 2010).

In this work, we build the surrogate dataset with the same power spectrum as in the original data, where the information component encoded in time is randomized, cf. **Figure 2A**. For each time index of the signal we randomly select a novel index value, such that all the index values are chosen exactly once (permutations without repetition). Then, we rearrange the timefrequency representation of trials according to the chosen index values and compute recurrence plots of surrogates by repeating steps (ii) and (iii) shown in **Figure 1**. This procedure is repeated S times per trial. **Figure 2A** illustrates how to obtain the surrogate set from T trials. Examples of an original RP and a corresponding surrogate RP are provided in **Figure 2B**.

We compare pixel-related statistical measures between the set of the original recurrence plots from different trials and their surrogates to determine whether original RPs preserve the common underlying signal dynamics in statistically significant way. This comparison is illustrated in **Figure 2C**. In detail, we denote the set of T recurrence plots obtained from the original trial data by {RP<sup>k</sup> } T k=1 and its surrogate set by {SRP<sup>k</sup> } S·T k=1 . In our simulations, there are T = 10 trials in total, where the number of surrogates generated per trial is S = 100. The full set of surrogates counts S ·T = 1000 surrogate RPs. At first, we perform pixel-wise statistical analysis tests between the corresponding pixels of the original and the surrogate recurrence plots. Let B¯ = {r {k} i,j } T k=1 be the vector that consists of the set of pixels with same coordinates in the original RPs and Bˆ = {r {k} i,j } S·T k=1 is the corresponding vector of pixel values for surrogates. Vectors B¯ and Bˆ consist of values from the set {0, 1}, since RP elements ri,<sup>j</sup> by definition take binary values, cf. Equation (1). To perform a chi-square test for categorical data, we build a two-by-two contingency table. For explanation, this tables first row takes values from the original RPs and the second row contains values from surrogate RP. The first table column marks the number of values ri,<sup>j</sup> = 1 and the second column the number of elements ri,<sup>j</sup> = 0. The elements of this table (two rows and two columns) have the coordinates (l, m), l, m ∈ {1, 2}. Then, the chi-square statistics for the pixel (i, j), i, j ∈ {1,... , N} is computed by

$$\chi^2(i,j) = \sum\_{l \in \{1,2\}} \sum\_{m \in \{1,2\}} \frac{\langle f\_o^{(i,j)}(l,m) - f\_e^{(i,j)}(l,m) \rangle^2}{f\_e^{(i,j)}(l,m)} \dots$$

Here, f (i,j) <sup>o</sup> (l, m) is the observed table value at the coordinate (l, m) for the pixel (i, j) and f (i,j) <sup>e</sup> (l, m) is its expected frequency. The latter value is computed as f (i,j) <sup>e</sup> (l, m) = nr(l)nc(m)/q, where nr(l) is the total number of elements in the row l, nc(m) is the number of elements in the column m and q is the total number of elements in the two-by-two table. The calculated chi-square value is compared with the result in the chi-square table for predefined values of the degree of freedom df = 1 and the significance level α<sup>s</sup> = 0.05. If the calculated chi-square value is larger than the value in the table, the hypothesis that signals share the same distribution is rejected, see Yates (1934) for more details. In this

Frontiers in Systems Neuroscience | www.frontiersin.org January 2016 | Volume 9 | Article 184 |

work, the outcomes of chi-square tests are visually represented as matrices A = [ai,j], i, j ∈ {1,... , N} whose elements take values

$$a\_{i,j} = \begin{cases} 1, & \text{if distributions of trial and surrogate sets} \\ & \text{are different,} \\ 0, & \text{otherwise.} \end{cases} \tag{5}$$

In this work all the figures follow the same color code as for illustrating recurrence plots, i.e., white pixels denote values ai,<sup>j</sup> = 0 and black pixels stand for ai,<sup>j</sup> = 1.

Since single elements in RPs are correlated to neighboring elements caused by the underlying dynamics, the underlying assumption of independent recurrence matrix elements does not hold and corrections of the significance test should be applied, such as the Bonferroni correction. To this end, in the examples of artificial datasets, we have performed a t-test which is based on the hypothesis that original and surrogate signals have the same distribution of mean values. The statistics are computed based on the pixels and their mean values in a 5 × 5 neighborhood around each pixel. In addition, we have applied a Bonferonni correction.

# 2.3. Datasets

To illustrate different analysis steps and to validate the power of the proposed method, we first apply the proposed algorithm to two artificial datasets. Then, the methodology is applied to experimental datasets. Single trials of these datasets are illustrated in **Figure 3** and their origin is described in detail below. For both artificial datasets, we model the tria-to-trial variability by a temporal shift of the data in time combined with additive measurement noise.

#### 2.3.1. Transient Oscillations

A modified Lotka-Volterra model with n = 3 interactive elements (Rabinovich et al., 2008a,b)

$$\frac{d\mathbf{x}\_i(t)}{dt} = \mathbf{x}\_i(t) \left(\sigma\_i - \sum\_{j=1}^n \rho\_{i,j} \mathbf{x}\_j(t)\right),\tag{6}$$

serves as an abstract model of event-related brain potentials (beim Graben and Hutt, 2015). Here xi(t) ≥ 0, i ∈ {1, 2, 3} is the activity rate of the element i, σ<sup>n</sup> is the growth rate of the n-th population and ρ<sup>i</sup> define interactions between elements. In our setup, σ<sup>1</sup> = 1, σ<sup>2</sup> = 1.2 and σ<sup>3</sup> = 1.6, ρii = 1, ρ<sup>12</sup> = 1.33, ρ<sup>13</sup> = 1.125, ρ<sup>21</sup> = 0.7, ρ<sup>23</sup> = 1.25, ρ<sup>31</sup> = 2.1, and ρ<sup>32</sup> = 0.83. The output signal s(t) is a linear superposition of transient oscillations with frequencies ν<sup>1</sup> = 170 Hz, ν<sup>2</sup> =20 Hz, ν<sup>3</sup> =75 Hz, where at one time instance, only one of these three components is dominant, see more details below. We point out that these frequencies are chosen rather arbitrarily for an optimal illustration. The activity rate x<sup>i</sup> defines the amplitude of the component a<sup>i</sup> with frequency ν<sup>i</sup> and the output signal obeys

$$s(t) = \sum\_{i=1}^{3} a\_i(t) \sin(2\pi \,\nu\_i t) + \xi(t) \quad , \quad a\_i(t) = e^{-(\chi\_i - \sigma\_i)^2 / 2\eta\_i^2}, \text{(7)}$$

with η<sup>1</sup> = 0.5, η<sup>2</sup> = 0.33, η<sup>3</sup> = 0.4. By this construction, the amplitudes a<sup>i</sup> increase and decrease in a certain time window outside of which they almost vanish. These windows of the three oscillation modes i = 1, 2, 3 do not overlap and the transitions between them are rather rapid. The variable ξ (t) represents measurement noise and its random values are i.i.d. Gaussian noise with zero mean and variance 0.5. The sampling rate is 450 Hz. We generate 10 trials which are time-jittered by shifting the trials by 1 sample to later instances, while each trial is subject to additive noise different in each trial. A single trial is given in **Figure 3A**.

#### 2.3.2. Lorenz dataset

The Lorenz system (Lorenz, 1963) is a well-studied threedimensional differential equation system

$$\frac{d\mathbf{x}}{dt} = -\sigma \mathbf{x} + \sigma \mathbf{y} \,, \, \frac{d\mathbf{y}}{dt} = \rho \mathbf{x} - \mathbf{y} - \kappa \mathbf{z} \,, \, \frac{d\mathbf{z}}{dt} = -\beta \mathbf{z} + \kappa \mathbf{y} \,.$$

with σ = 10, ρ = 28, β = 8/3. Its solutions show nontrivial transient dynamics and their wings represent metastable states as explained above (cf. beim Graben and Hutt, 2013). We study the univariate time series x(t), which is the solution of the above given system of equations. This time series may represent a macroscopic measured signal such as EEG recording (Skarda and Freeman, 1987; Basar, 2006), capturing activity from different metastable sources. The sampling rate is equal to 2100 Hz. We generate ten signal trials time-jittered by shifting the signal by 1 sample to later instances and add i.i.d. zero mean Gaussian noise with unity variance to the signals. One trial signal is illustrated in **Figure 3B**.

## 2.3.3. Ferret Dataset

The experimental dataset under study in the present work are Local Field Potential (LFP) measurements collected as described in Sellers et al. (2013, 2015a,b). Briefly, female ferrets were anesthetized, intubated, and underwent surgery to gain access to primary visual cortex (V1, ∼3 mm anterior to lambda and 9 mm lateral to the midline). Anesthesia induction was achieved with an intramuscular injection of ketamine (30 mg/kg) and xylazine (1–2 mg/kg), and anesthesia maintenance was achieved with 1.0% isoflurane (10–11 cc, 50 bpm, 100% medical grade oxygen), with continuous IV infusion of xylazine (1.5 mg/kg/h xylazine with 4.25 mL/h 5% dextrose lactated ringer's). Animals were head-fixed in front of the presentation screen and a 32 channel depth probe was acutely inserted into cortex (50 microns contact spacing along the z-axis, NeuroNexus, Ann Arbor, MI) and was positioned to cover all cortical layers. The reference

electrode was located on the same shank (0.5 mm above the top recording site) and was positioned in 4% agar in saline above the brain. The full-field visual stimulus was presented on a 52 × 29 cm monitor with 120 Hz refresh rate and full high-definition resolution (1920 × 1080 pixels, GD235 Hz, Acer Inc, New Taipei City, Taiwan) at 47 cm distance from the animal. Each trial was 30 s long and consisted of three parts: (i) recording interval [0 − 10)s is a baseline (screen is black); (ii) t<sup>s</sup> ∈ [10 − 20) s is the presentation of the sine-wave luminance gratings; (iii) [20−30) s is "post-baseline" (screen is again black). Visual stimuli were interleaved with other types of stimuli (all in randomized order), for instance with a black screen or with a strongly spatially filtered image of foxes (foxes are natural enemies of ferrets), see **Figure 4**.

The sine-wave luminance grating was presented at a rate of 1 Hz for 10 s (during each 1 s period, progressive frames transitioned from black to white to black and all the screen pixels had the same color for any given frame). In the subsequent analysis, we consider a subset of recordings. The dataset under study starts 0.5 s before stimulus onset and lasts until 3 s at the end of the third stimulus cycle. This stimulus is a black screen at t = 0, 2, and 3 s with luminance maxima at t = 0.5, 1.5 s, and 2.5 s. A single trial is illustrated in **Figure 3C**.

Electrophysiological recordings were conducted during stimulus presentation. Unfiltered signals were first amplified with MPA8I head-stages with gain 10 (Multichannel Systems, Reutlingen, Germany), then further amplified with gain 500 (Model 3500, A-M Systems, Carlsborg, WA), digitized at 20 kHz (Power 1401, Cambridge Electronic Design, Cambridge, UK), downsampled to 1 kHz afterwards, and digitally stored using Spike2 software (Cambridge Electronic Design). In total, 20 trials across two sessions conducted on different days were analyzed, the session sets of 10 trials are called session one and session two in the following. Datasets are downsampled to the sampling rate equal to 100 Hz. All procedures were approved by the University of North Carolina-Chapel Hill Institutional Animal Care and Use Committee (UNC-CH IACUC) and exceed guidelines set forth by the National Institutes of Health and U.S. Department of Agriculture.

# 3. RESULTS

In this section, we first apply the proposed method to two artificial datasets to verify if it reveals dynamics given the noisy set of trials. After validation of the method, we apply it on the experimental dataset (ferret dataset) and study whether it well extracts the dynamics from the recorded trials.

To understand the results, we first shortly describe recurrence plots for several simple test signals. As previously mentioned, black pixels denote recurrence events and white ones its absence. All recurrence plots have a black diagonal line, by definition (see Equation 1). Signals without any recurrence have a white square RP with a black diagonal line. Random noise signals have a random distribution of black pixels in the plot, with the exception of the black diagonal line. A simple periodic signal has a recurrence plot that consists of the black diagonal line and other black lines that are parallel to the diagonal, where the distance between them will reveal the period of the signal. More complex signals that have recurrent states may show different structures in RPs, for example, checkerboard-like patterns. These black colored fields, to which we refer as to recurrence domains, may have different sizes and shapes. For two artificial datasets we expect to observe repetitive black patterns that correspond to repetitive states within signal components. For the experimental dataset, we expect to observe recurrence patterns that are directed by the onset of the visual stimulus.

# 3.1. Artificial datasets

We demonstrate our methodology in **Figure 5**, which shows the analysis steps for the examples of transient oscillations (**Figures 5A–D**) and the Lorenz attractor (**Figures 5E–H**).

The time-frequency representation of one transient oscillation trial is shown in **Figure 5A**. As previously mentioned in Section 2.3.1, the corresponding signal exhibits three periodic components. We visually inspect the figure and observe high power spectrum values around the following time windows: (i) {(0, 11),(42, 54),(85, 97)} ms around ν<sup>1</sup> = 170 Hz (dark red horizontal line segments); (ii){(19, 26),(64, 72)} ms around ν<sup>2</sup> = 20 Hz (broad orange areas); (iii) {(24, 38),(74, 87)} ms around ν<sup>3</sup> = 75 Hz (dark red horizontal line segments). Note that for other trials these values may fluctuate because the frequency and time window of the current active component vary due to noise that models trial-to-trial variability. **Figure 5B** shows recurrent blocks (in black) in a single trial which correspond well to the dynamics observed in the data, cf. **Figures 5A**, **3A**. For explanation, these recurrence blocks correspond to MSs and the white parts represent transients between them.

The time-frequency representation of one Lorenz attractor trial is given in **Figure 5E**. The approximate time intervals during which the system stays in each of the two wings are visually inspected from the power spectrum values. For the wing in time intervals {(0, 30),(90, 100)} ms, **Figure 5E** shows a peak at ∼30 Hz corresponding to the oscillation frequency in the Lorenz wing, see **Figure 3B**. The other wing is reached in the time intervals {(40, 60),(65, 80)} ms in accordance to the power peak at about 40 Hz. Note that for other trials time intervals may be different due to varying trials in the set. **Figure 5F** shows recurrent blocks in a single trajectory. The recurrence blocks repeat in the correct time windows and represent the different wings, i.e., the MSs.

Time-frequency representations of single trials are the basis for the recurrence analysis leading to recurrence plots given in **Figures 5B,F** for the respective datasets. These plots show the metastable dynamics of the transient oscillations and the Lorenz trajectories in the corresponding time windows as recurrent structures. The recurrent, i.e., repetitive, structure is visible in the illustrated trial of the corresponding data. Now, considering several trials these recurrent structures may vary due to the trial-to-trial variability. Nevertheless, to study the recurrent structure common to all trials, we employ the statistical inference method and extract statistically significant areas of recurrence plots, as shown in **Figures 5C,D,G,H**. The recurrent structure is obvious in these plots, reflecting the underlying recurrence structure in the artificial signals. In addition, these results demonstrate that the methodology extracts recurrence structures common in several trials, although the recurrent structure is less obvious in single trials, **Figures 5B,C**. **Figures 5D,H** show the multiple comparison-test results for both artificial datasets. The white area increases and the black areas are more focussed on the red squares, i.e., spurious recurrences (black dots) are removed and and separated well from transient (white areas). Hence the multiple-comparison test improves the statistical inference.

We point out again, that the extraction of the recurrent structure from the univariate data shown in **Figure 5** is possible only by the spectral power embedding, i.e., the transformation of the univariate data into multivariate data. The subsequent preliminary statistical inference allows to identify the recurrent

MSs which are common in all trials with a confidence of 0.95. By virtue of the spectral power embedding, the method permits to select certain frequency bands to study recurrence structure in specific frequency bands. This new element renders the spectral power embedding more flexible and hence superior to previous embedding techniques, such as the delay embedding based on Takens theorem. To illustrate this, the subsequent section shows results from experimental data in different frequency bands.

# 3.2. Experimental Data

After studying artificially generated trials and verifying that the proposed method extracts well the dynamics features given by repetitive black structures, we now investigate whether such structures can be found in the experimental data as well.

**Figure 6** provides the time-frequency representations of two single trials of the same session **Figures 6A,C** and the corresponding recurrence plots **Figures 6B,D**. We observe a high trial-to-trial variability between both trials. This can be observed both in the time-frequency representations and the resulting recurrence plots. For instance, **Figure 6B** shows a single recurrent state in the data except in the time window during the first stimulus at t ∈ [0.25 s; 1 s]. Hence the system remains close to the resting state (t < 0) during the first stimulus. Conversely, **Figure 6D** reveals that the baseline activity, i.e., activity before stimulation in the time interval [−0.5 s; 0 s], recurres in the interval [2.5 s; 3 s]. In addition, the activity at about t = 2 s resembles the activity just after t = 2.5 s. These different findings for two trials are surprising since the experimental presentation of the visual stimulus is well-controlled and the stimulus is simple enough to expect almost identical neural responses.

To reveal the recurrent structure that is common in all trials, we now study the trial-to-trial variability of recurrence plots and aim to reveal whether the signal trials preserve the same dynamical behavior, cf. **Figure 7**. Applying the statistical method, we investigate the similarities of the results obtained from ten trials measured by a single granular sensor and from the set of 10 averaged signal trials, where the average is taken over eight granular layer sensors. This analysis is done for both ferret datasets. Moreover, we detail the analysis considering particular frequency bands which are of interest for anesthesia. To this end, we compute recurrence plots using the values of the power spectrum coefficients in the corresponding frequency bands as illustrated in **Figure 1**.

**Figure 7** shows the statistically significant parts of the recurrence plots for the δ- and the α-frequency band and for all frequencies (chi-square and t-test results). The figure reveals that there is no statistically significant recurrent structure in the δ frequency band in signals under study. Conversely, the α-frequency band exhibits significant recurrent structures in the single granular electrode in both datasets, cf. **Figures 7A,B**. For instance, in session one the first response to the stimulus at t = 0 s returns at t = 1 s. Results for all frequency bands differ to results obtained in the α frequency band. The differences are dependent on the experimental sessions suggesting the presence of strong recurrences in bands different to α and δ or strong noise artifacts. To gain further insights into the dependence on frequency bands, we consider single trials which represent spatial averages of time series from adjacent granular layers. This average denoises the time series. **Figures 7C,D** shows the corresponding results. **Figures 7C,D** show results from data in both experimental sessions revealing a similar recurrence structure now. Considering all frequency bands yields recurrences similar to the one obtained in the αband. Specifically, the prominent cross-shaped structure located at t = 0.5 s indicates a MS common to all data with t ≥ 0.5 s. Additional recurrences occur in the time intervals [−0.5; 0],

[1.7; 2.2], and [2.7; 3.0] ms. These results are consistent in the chi-square significance tests and the t-test involving corrections for multiple comparison. At last, we mention the prominent lack of recurrence in the baseline time interval observed in a single electrode in session one, cf. **Figure 7B**. Since it does neither occur in session one nor in the spatially averaged data shown in **Figures 7C,D**, it appears to be spurious and is neglected.

The experimental paradigm includes visual presentations of stimulus types in a randomized order. The previous paragraphs show neurophysiological responses to the sine-wave stimulus only. To gain further insight into the trial-to-trial variability subject to various pre-stimuli, we have selected two subsets of sine-wave trials that have two different preceding stimuli, namely the "black screen" subset denoted by subset one and the subset of "strongly spatially filtered version of foxes" denoted by subset two. Subset one includes the trials {4, 6, 9}, while subset two is composed of the trials {1, 5, 10} of datasets session one and session two. The comparison of various pre-stimuli data is done by the chi-squared difference measure based on recurrence plots of both subsets. **Figure 8** shows the statistically significant recurrences that are common in stimulus responses on both types of pre-stimuli. The diagonal lines are absent from figures, which suggests that at each time instance two comparison signals differ. Poor but visible recurrent structures in δ-band are grouped into two distinct blocks which distinguish the activity before the stimuli (around t = 0 s) and during the stimuli, for t ∈ (1, 3) s. In α-band, the figure shows more prominent recurrences, such as the patterns around t = 1.5 and 2.7 s. We point out that recurrences within δ- and α-bands do not overlap, except in the pre-stimuli period, for t ∈ {−0.5, 0}s. Finally, considering all the frequency bands together does not reveal significant similarities of two pre-stimuli. Results from the chi-square test and the t-test involving multiple comparison correction are similar. However, it is interesting to note that the ttest reveals more significant common recurrences than obtained with the naive chi-square test.

# 4. DISCUSSION

The present work introduces a new recurrence analysis methodology for univariate time series. The first new element is the transformation to a time-frequency representation leading to a multivariate time series of spectral power. This new technique generates a new high-dimensional phase space in which the instantaneous power of the signal evolves. This highdimensional phase space is mandatory to apply recurrence analysis. In addition, it permits to compute recurrence plots for specific frequency bands. The second new element is the statistical analysis of recurrence plots that takes into account spurious recurrence structures and allows to suppress them. The combination of the two methods permits to extract temporal recurrence structures in data which may reflect underlying transient dynamics in a certain range of frequencies that would have been hidden in conventional methods. To our best knowledge these two techniques have not been considered before.

The first results for two artificial datasets illustrate the methodology and indicate that method detects recurrences in a variable dataset (noise-induced trial variability) by the statistical analysis as seen in **Figure 5**. These results on artificial datasets prove that the method reveals underlying recurrences in a set of trials if they are present in these trials.

prominent lack of recurrence in the α−band.

The subsequent analysis of single Local Field Potentials measured experimentally in ferret visual cortex reveals a high trial-to-trial variability, cf. **Figure 6**. The trial-to-trial variability is surprising due to the well-controlled experiment revealing an intrinsic ongoing activity (Arieli et al., 1995). This result demonstrates that it is mandatory to take into account recurrence variability in several trials. This is done by the methodology proposed. Detailed recurrence analysis of specific frequency bands in **Figure 7** reveals missing recurrences in the δ band whereas α-activity exhibits statistically significant temporal recurrence. This important finding reflects a fundamental difference of the nature of δ- and α-activity which has been shown in previous experimental studies on the neural origin of both signal features (Alkire et al., 2000; Ching et al., 2010; Hashemi et al., 2014). Our results suggest that the brain may decode information processing steps in different frequency bands. This might be of importance in previous studies and may shed some new light on neural processes, such as on metastable states in EEG during the emergence from unconsciousness (Hudson et al., 2014) and metastable states in bird songs (Yildiz and Kiebel, 2011).

The effects of pre-stimuli have been hypothesized (Van Rullen et al., 2011; Lundqvist et al., 2013) and we have investigated the effect of pre-stimuli. The performed analysis is based on a rather small set of trials reflecting the responses to identical stimuli. To have sufficiently large dataset for tests, we merged trials coming from two recording sessions. We note that trials coming from two sessions may not be independent, which may introduce errors. We found negligible effects in the δ frequency bands but differences in the temporal recurrence structure in the α frequency band. This result indicates that α-activity is more sensitive to pre-stimuli than δ-activity in the experimental setup under study. This finding is in full line with previous theoretical (Lundqvist et al., 2013) and experimental (Romei et al., 2008) studies on the importance of phase and power of prestimulus α-activity. In addition, we notice the absence of the diagonal line and other strong recurrence patterns visible in **Figure 7**. This may be the result of merging trials from different sessions, which was necessary to obtain larger test set for the analysis.

The present work shows that trial-to-trial variability in neurophysiological data occurs in spite of well-controlled

and simple response-driven experimental conditions and demonstrate how to extract recurrent structures nevertheless. The methodology proposed makes it necessary to choose a well-adapted technique to transform the univariate times series to a multivariate time frequency signal. In addition to our current choice of a spectral reassignment technique, we have employed a conventional wavelet technique using complex Morlet mother wavelets and performed the same recurrence analysis (results are not shown). It turns out that this conventional method does not provide high-quality extraction of transient recurrent structures, given by the reassignment method. This may result from the worse time-frequency resolution of conventional Morlet wavelets. Future work will further investigate the best choice of multi-resolution timefrequency methods. Moreover, the methodology considers surrogate data generated by a temporal random shuffling of data and hence destructing all temporal structure. Future work may include the destruction of the recurrence structure by phase randomization in certain frequency bands (Li et al., 2010).

To conclude, in this work we propose a novel analysis method for trial-to-trial variability of recurrence plots in univariate time series applying a novel statistical analysis technique. This extension of recurrence analysis by a statistical technique is motivated by the fact that many physiological datasets have a limited number of trials but posses the intrinsic recurrence property of patterns of interest. Inspired by the fact that particular physiological patterns very often occur in specific frequency bands, we first build novel recurrence plots from a time-frequency signal representation. A low dimensional timefrequency signal that is built by the band median filter is then used to obtain original trial recurrence plots. Next, we use a chi-squared statistics to obtain statistically important areas of recurrence plots. The work reveals a strong trial-totrial variability of recurrences in experimental data in spite of the well-controlled experimental paradigm. Moreover, it turns out that recurrences occur in the α-frequency band, whereas activity in the δ-frequency band does not exhibit a temporal recurrent structure indicating frequency-dependent metastable states.

# 5. DATA SHARING

We provide the time-series of the transient oscillation dataset and Lorenz dataset on the webpage of the corresponding author (https://sites.google.com/site/tamtos/datasets).

# AUTHOR CONTRIBUTIONS

The majority of the analysis steps and the implementations have been performed by TT. KS and FF have provided the experimental data and neurophysiological insights. MF has contributed additional time-frequency analysis results and PB and AH have conceived the study. All authors wrote the manuscript together.

# ACKNOWLEDGMENTS

Research reported in this publication was supported by the National Institute Of Mental Health of the National Institutes

# REFERENCES


of Health under Award Number R01MH101547. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. AH acknowledges funding from the European Research Council for support under the European Union's Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no. 257253. PbG acknowledges support by a Heisenberg Fellowship of the German Research Foundation DFG (GR 3711/1-2).

Hutt, A. (2004). An analytical framework for modeling evoked and event-related potentials. Int. J. Bifurcat. Chaos 14, 653–666. doi: 10.1142/S0218127404009351


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Toši´c, Sellers, Fröhlich, Fedotenkova, beim Graben and Hutt. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Improved discriminability of spatiotemporal neural patterns in rat motor cortical areas as directional choice learning progresses

Hongwei Mao<sup>1</sup> , Yuan Yuan<sup>1</sup> and Jennie Si 1, 2, 3 \*

*<sup>1</sup> Electrical Engineering, School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USA, <sup>2</sup> Graduate Faculty of the School of Biological and Health Systems Engineering, Arizona State University, Tempe, AZ, USA, <sup>3</sup> Affiliate Faculty of the Interdisciplinary Graduate Program in Neuroscience, Arizona State University, Tempe, AZ, USA*

#### Edited by:

*Daniel Durstewitz, Heidelberg University, Germany*

#### Reviewed by:

*Bruno B. Averbeck, National Institute of Mental Health, USA James M. Hyman, University of British Columbia, Canada Brice Bathellier, Centre National de la Recherche Scientifique, France*

#### \*Correspondence:

*Jennie Si, Electrical Engineering, School of Electrical, Computer and Energy Engineering, Arizona State University, 650 E Tyler Mall, Tempe, AZ 85287, USA si@asu.edu*

> Received: *16 October 2014* Accepted: *16 February 2015* Published: *06 March 2015*

#### Citation:

*Mao H, Yuan Y and Si J (2015) Improved discriminability of spatiotemporal neural patterns in rat motor cortical areas as directional choice learning progresses. Front. Syst. Neurosci. 9:28. doi: 10.3389/fnsys.2015.00028* Animals learn to choose a proper action among alternatives to improve their odds of success in food foraging and other activities critical for survival. Through trial-and-error, they learn correct associations between their choices and external stimuli. While a neural network that underlies such learning process has been identified at a high level, it is still unclear how individual neurons and a neural ensemble adapt as learning progresses. In this study, we monitored the activity of single units in the rat medial and lateral agranular (AGm and AGl, respectively) areas as rats learned to make a left or right side lever press in response to a left or right side light cue. We noticed that rat movement parameters during the performance of the directional choice task quickly became stereotyped during the first 2–3 days or sessions. But learning the directional choice problem took weeks to occur. Accompanying rats' behavioral performance adaptation, we observed neural modulation by directional choice in recorded single units. Our analysis shows that ensemble mean firing rates in the cue-on period did not change significantly as learning progressed, and the ensemble mean rate difference between left and right side choices did not show a clear trend of change either. However, the spatiotemporal firing patterns of the neural ensemble exhibited improved discriminability between the two directional choices through learning. These results suggest a spatiotemporal neural coding scheme in a motor cortical neural ensemble that may be responsible for and contributing to learning the directional choice task.

Keywords: associative learning, action selection, agranular medial and lateral areas, plasticity, support vector machines

# Introduction

When selecting an action among alternatives in response to an external stimulus, an animal usually makes its choice according to consequences of the actions taken. Animals choose those actions that have resulted in rewards in the past and thus, learning takes place by correctly associating a stimulus with an appropriate response. A neural network that underlies the acquisition of this stimulus-response association has largely been identified (Murray et al., 2000), and it points to the prefrontal cortex (PFC) and the basal ganglia as two key nodes for solving an associative learning task (Pasupathy and Miller, 2005). Within the frontal lobe, a rostro-caudal hierarchical organization supporting cognitive control functions such as action selection has been hypothesized (for a review see Badre, 2008). Primate studies have shown that premotor regions are also involved in learning and holding stimulus-response representations under the influence of prefrontal regions through topdown control (Koechlin et al., 2003; Boettiger and D'Esposito, 2005; Fluet et al., 2010). Additionally, the primary motor cortex has been suggested for encoding information beyond movement kinematics (Carpenter et al., 1999; Matsuzaka et al., 2007) such as features of visual stimuli that are behaviorally relevant (Zach et al., 2008; Eisenberg et al., 2011). Furthermore, the motor cortex is highly plastic for learning sensory-motor associations (Sanes and Donoghue, 2000). Putting it all together, the motor cortical regions in the frontal cortex are implicated for learning which action to select according to stimulus-response association.

Two types of adaptation could co-exist during sensorimotor association learning: motor skill learning that improves the execution of motor responses and associative learning that links sensory cues with specific response actions (Cohen and Nicolelis, 2004). Motor skill learning alone could induce neural plasticity ranging from synaptic connections (Xu et al., 2009), changing neural firing rates (Li et al., 2001; Kargo and Nitz, 2004; Rokni et al., 2007), to the motor map (Kleim et al., 1998, 2004) in both young and adult motor cortices. Therefore, motor skill learning could become a confound factor when studying sensorimotor association learning and should be treated with care. Aside from well-studied motor skill learning, whether and how motor cortical activity would adapt during associative learning is still unclear and requires further investigation.

In a previous study (Cohen and Nicolelis, 2004), rats learned to associate directional movements in response to either a high or a low tone. Significant neuronal firing rate changes in the primary motor cortex were observed on the first day when an animal's movement skill improved, but not in the following 2 days when movement parameters were stable and associative learning dominated. In some other studies where animals learned to respond to external sensory cues with appropriate actions, learning-related neural dynamics were evident in motor cortical neural ensemble activity patterns (Laubach et al., 2000; Huber et al., 2012). Based on these results we hypothesize that neural adaptation induced by learning sensorimotor associations would be reflected in changes in spatiotemporal neural firing patterns in motor cortical areas. To test this hypothesis, we had rats learn to perform a directional choice task. The goal of the task was to make a left or right side lever press in response to a left or right side light cue, respectively. Single units were recorded from rat's medial agranular (AGm) and lateral agranular (AGl) areas. Spatiotemporal neural firing patterns were investigated using support vector machines. Improved discriminability in neural patterns was observed as learning progressed.

# Materials and Methods

# Animal Handling and Surgery

All procedures were in accordance with guidelines of the National Institutes of Health and approved by the institutional Animal Care and Use Committee at Arizona State University. Rats (Long-Evans, male) arrived at the age of about 2 weeks weighing around 50 grams and were handled daily by experimenters to get accustomed to the environment. They started pre-training after reaching 200 grams to master the motor skill of lever pressing, which only involved pressing a single lever (no choice) in response to a light cue above the lever. The pre-training apparatus was similar to that used for recording to help familiarize rats with the recording environment. After achieving a behavioral accuracy of 90% or above for at least 3 consecutive days on the pre-training task, and once their weight reached 400 grams, rats were implanted with a chronic electrode array.

For electrode implant surgery, rats were anesthetized by an intramuscular injection of KXA (10 mg/ml ketamine, 2 mg/ml xylazine, and 0.1 mg/ml acepromazine; 0.1 ml/100g), shaved in the incision area, and placed in a stereotaxic frame. A heated water blanket was used to maintain rat's body temperature at around 35◦C. Rat's heart rate and oxygen level were monitored throughout surgery with a pulse oximeter. KXA updates (0.05 ml/100g) were administrated approximately every hour during surgery after the initial shot. Craniotomy was performed over the AGm and AGl areas of the left hemisphere of the rat brain. A microwire array was centered at 2 mm lateral and 3 mm rostral from the bregma (**Figure 1C**), and lowered about 1.8– 2.3 mm underneath dura, aiming for layer V pyramidal neurons. An acrylic head cap was formed to support the electrode array. The head cap was fixed to the skull with three screws. A subcutaneous injection of 0.1 ml meloxicam was given for pain relief after surgery, and three more shots were given for the following 3 consecutive days. The rats had 7–10 days or as needed to recover before they were food restricted for recording sessions.

# Behavioral Task

Rats were freely moving in the recording chamber, and self-paced to start a trial by pressing the retractable center ready lever. One of the five cue lights (from left to right: LL, L, C, R, and RR) would appear (**Figure 1A**) immediately upon ready LP. The left and right response levers would extend 2 s after cue light onset (**Figure 1B**). Pressing the left lever once would "move" the light one position to the right and pressing the right lever would "move" the light to the left. Once the light cue reached and then remained at the center position for at least 1 s, the trial ended as a success. Otherwise a trial was considered a failure if the light cue ended up at any position other than the center. A feedback tone was played immediately upon the end of a trial: a low frequency tone of 1 kHz in case of a success and a high frequency tone of 12 kHz in case of a failure. A sugar pellet reward was delivered 0.5 s after the feedback tone for a successful trial. The inter-trial interval was 8 s for successful trials and 15 s for failed trials. The five cues were presented in a pseudo-random fashion with equal probability of presence.

#### Recording Sessions

After rats recovered from surgery, daily recording sessions began, each of which lasted about 60 min. Rats were food restricted during the recording period while the body mass was closely monitored.

The implanted electrodes were arranged in a 2 × 8 matrix, with 500 or 375µm row separation, and 500µm electrode spacing. The polyimide-isolated tungsten microwires were 50µm in diameter and 5 mm in length. The electrode tips were cut at a sharp 60◦ angle (TDT Inc., FL). A total of 16 channels of raw waveforms were recorded simultaneously using a RX5 Pentusa Base Station or a RX7 Microstimulator Base Station (TDT Inc., FL). Neural signals picked up by electrodes were passed to a unity gain preamplifier (bandpass 2.2 Hz ∼ 7.5 kHz) through an Omnetics or a ZIF-Clip headstage, and then sampled and stored at 24.414 kHz by the base station.

Rat behavior while performing the task was monitored and recorded using cameras (25 fps). Rat head position was determined offline by the implanted head cap as well as left and right ear positions. The head position was tracked and extracted to indicate rat movement trajectory, which was calculated for left side and right side movements separately over recorded trials. Variation in movement trajectory was obtained as the distance between a given movement trajectory during a trial and the mean trajectory in a single recording session.

#### Spike Sorting

Action potentials were detected and classified off-line using our own M-Sorter software (Yuan et al., 2012), which is based on the multiscale correlation of wavelet coefficients (MCWC) detection algorithm (Yang et al., 2011). The M-Sorter has been tested and compared with two popular sorters: the Wave Clus and the automatic mode of Offline Sorter by Plexon (T-Distribution EM method). The M-Sorter consistently outperformed or was at least comparable to the compared sorters (Yuan et al., 2012). One isolated unit with highest firing rate was extracted from each of the electrodes. Experimenters also inspected spike waveforms, inter spike intervals, and other measures to ensure the quality of single unit clusters (**Figure 1D**). According to the sites of implanted electrodes, recorded neurons were in the AGm and AGl areas of the rat frontal cortex (Paxinos and Watson, 2005) involving forelimb, neck, and vibrissae areas (Neafsey et al., 1986; Remple et al., 2001). Intracortical microstimulation was also performed to confirm implant electrode location.

For each rat, only those electrodes that consistently picked up unit action potentials in all sessions were included in neural activity analysis. By doing so, we were able to analyze neural ensembles of the same size over sessions to make the results comparable. The analyses in this study as described below were based on neural ensembles. Therefore, we did not require tracking same neurons over learning sessions (Laubach et al., 2000; Cohen and Nicolelis, 2004).

### Firing Rate Modulation

In this study, L-L trials are used to denote those trials in which rats reported left side choices by pressing the lever on the left side in response to left side cues, and similarly we define R-R trials. Single unit firing rates in single trials were calculated using a 100 ms data window sliding at 20 ms steps (50 bin/s) through the cue-on task period (**Figure 1B**). The mean firing rate for a data window (CO1, CO2, CO3, or CO) was the average of all binned firing rates in the respective data window.

Let the ith neuron's mean firing rate in session k for all L-L and R-R trials be denoted as M<sup>i</sup> L (k) and M<sup>i</sup> R (k), respectively. The ith neuron's mean firing rate was then defined as Mi (k)= 1 2 (M<sup>i</sup> L (k)+M<sup>i</sup> R (k)). The ith neuron's firing rate difference between L-L and R-R trials in session k was calculated as D i (k)= M<sup>i</sup> L (k)−M<sup>i</sup> R (k).

The ensemble mean firing rate M(k) of N isolated units in session k was the average over all recorded trials of isolated units, i.e., M(k)= 1 N P<sup>N</sup> <sup>i</sup> <sup>=</sup> <sup>1</sup> M<sup>i</sup> (k).

The ensemble mean firing rate difference between L-L and R-R trials was then the average of the absolute value of single unit rate differences, i.e., D(k)= 1 N P<sup>N</sup> i = 1 D i (k) . As such, each recording session corresponded with one measurement for the ensemble mean rate and another measurement for the ensemble mean rate difference.

In addition to firing rates, firing variability was also monitored. First, we calculated the standard deviation of firing rate of unit i in session k, S i L (k) and S i R (k), for L-L and R-R trials, respectively. Then, the mean standard deviations, SL(k) and SR(k), were calculated as the average across units, respectively.

To study how the ensemble mean firing rate and ensemble mean rate difference would change during learning from session to session, linear regressions were performed against normalized session numbers (between 0 and 1). The sign of the regression line slope was determined according to its confidence interval. A positive slope corresponded with increased rate measures while a negative slope with decreased rate measures. No change in rate measures was associated with a regression line slope that was not significantly different from zero. Similar linear regression analysis was used to examine changes of other measurements as described below.

In order to summarize results of multiple rats, the ensemble mean firing rates of single sessions were Z-scored (zero mean and standard deviation equal to one) over sessions for each rat. Then Z-scored ensemble mean rates from all rats were pooled together for linear regression analysis. Other measurements, including ensemble mean rate difference, mean standard deviation of firing rate, and SVM classification results as described below, were processed in a similar manner when their trends over sessions were explored by summarizing multiple rats' data.

#### SVM Classification of Neural Representations

We modeled neural firing patterns of L-L and R-R trials by training linear kernel support vector machines (SVMs). The input to the SVMs was spatiotemporal neural firing activity in the cue-on task period of a single trial while the output of the SVMs was the directional choice of left or right. All analyses were performed using customized Matlab programs (Mathworks Inc., MA).

SVMs solve a binary classification problem by determining a separating hyperplane with a maximized margin between two classes (Burges, 1998). Once the separating hyperplane is found, an SVM makes a classification decision for a given data sample <sup>P</sup> <sup>x</sup> according to the value of the decision function: df(**x**)<sup>=</sup> i αiK(**si**, **x**)+b, where support vector **s<sup>i</sup>** , weight α<sup>i</sup> and bias b are determined in the training process automatically once input and output data are presented for training, and the kernel function K is a dot product in case of a linear kernel. If df(**x**)≥ 0, **x** is classified as an L-L trial, otherwise it is classified as an R-R trial. The decision function value could be interpreted as the distance from the sample point to the separating hyperplane. The greater this distance the less ambiguous the final classification.

In our analysis, a 1500 ms data window in the cue-on period (CO: 300 to 1800 ms after cue onset, **Figure 1B**) was used. This window was divided into three non-overlapping 500 ms time bins. Spike counts in these bins formed one vector representation for each spike train of each unit. Spike count vectors of simultaneously recorded units were then concatenated to form a spike count vector representation of the recorded neural ensemble (**Figure 2**). Thus, there was one ensemble vector or one data sample for each trial, and SVMs were trained based on data samples from both classes (L-L and R-R trials) in each recording session for each rat.

To obtain statistically representative results, a total of 100 SVM classifiers were trained and tested for each session. In each of the 100 classifiers, a constant number of trials were randomly chosen from both L-L and R-R classes. Specifically, 20% of the randomly chosen trials from each class formed the test set, and the remaining 80% formed the raw training set, which was further

FIGURE 2 | Data preparation for SVM decoding. Spike trains of all simultaneously recorded single neurons from one task trial formed one data sample where the spike counts in non-overlap bins (e.g., 500 ms bins) were concatenated to form a spike count vector. Two classes of data samples from L-L and R-R trials composed the data set.

processed as follows before training an SVM. First, five trials of the same class were randomly selected from the raw training set, each with a respective spike count vector. Then an average spike count vector was obtained based on the five spike count vectors, and was used as an input training data sample for the SVM model. A test data sample simply was a single spike count vector of a test trial. The SVM performance measure was based on averaged test set classification accuracy from the 100 SVM classifiers. This procedure was repeated for each recording session over the entire directional choice learning process. SVM based classification performance of L-L vs. R-R trials over multiple sessions were then inspected using linear regression.

As a control, SVM classification analysis was also performed using a 500 ms time window around response lever press (LP, −100 to 400 ms around the press, **Figure 1B**). Data was prepared in a similar way as described above and 100 ms time bins were used to compute spike counts. To make classification performance comparable, three 500 ms windows within the cue-on period were selected (CO1, 500 to 1000 ms after cue onset; CO2, 1000 to 1500 ms; CO3, 1500 to 2000 ms; **Figure 1B**). SVM classification analysis was repeated in these data windows and compared with that using data in LP.

Note that in our analysis, 100 ms bin size was used for 500 ms data window (CO1, CO2, CO3, and LP) based direction predictions by SVM. But for SVM classification analyses where the CO window (**Figure 2**) was involved, 500 ms bins were used to form spike count vectors.

# Results

#### Behavioral Results

Male Long-Evans rats (n = 9) started learning the directional choice task by trial and error from a naïve state. Behavioral accuracy in each recording session was monitored and calculated as the number of correct trials over the total number of trials in that session. Rats gradually improved the accuracy over sessions, from 30.8% (average, range from 14.1 to 47.3%) in session 1 to 76.0% (average, range from 55.3 to 93.4%) in session 18 (rank-sum test, p < 0.001; **Figure 3A**). Linear regressions of behavioral accuracies vs. session numbers revealed that seven rats significantly improved their performance except rat A09 and I10. Actually rat A09 didn't learn the right side choices, and rat I10 struggled with both left and right side choices. Among the seven rats, one of them (J11) reached 75% accuracy, all the other six rats went above 80%, and two rats (W09 and O10) even achieved over 90% accuracy. We therefore used data from the seven rats when reporting results against normalized session numbers as learning progressed. When results are based on data from all nine rats, it will be specified accordingly.

Once the rat self-started a new trial (**Figure 1B**), he had 2 s to choose from the two response levers prior to their extension. Upon response lever extension, he could make a press of his choice within 1 s. In this analysis, the response latency was calculated as the time from response lever extension to the first press on the chosen lever. This latency decreased from 0.44 ± 0.16 s in

interface panel. Thin traces: trajectory records from single trials. Bold curves: mean trajectory traces of left side and right side movements. (D) Variation in movement trajectory (distance between the trajectory trace of a single trial and the mean trajectory) decreased during initial learning sessions.

session 1 to 0.32 ± 0.14 s in session 4 (mean ± STD; ANOVA, p < 10−<sup>5</sup> ; **Figure 3B**; nine rats). After the first three learning sessions the response latency became stabilized, and from the fourth session onwards, the latency measurements over sessions were not significantly different (linear regression slope at −0.0021 s/session, 99% confidence interval of [−0.0058 0.0016]). The reduction in response latency is an indicator of improved act of lever pressing.

During task performance, rats usually made quick movements toward their chosen lever right after light cue onset and their movements became stereotypical in a few sessions. Video analysis of rat movement trajectory during the cue-on period (**Figure 3C**) confirmed this observation. Typically, rats started moving in their chosen direction shortly after cue onset. Their directional movements were completed by about 1 s after cue onset and then rats stayed in front of the response lever waiting for lever extension. To measure differences in rat directional movement from trial to trial, variation in movement trajectory between a singletrial trajectory and the mean trajectory of a session was calculated for each session. As shown in **Figure 3D**, variations in movement trajectory decreased during the first three sessions and remained stable afterwards. Additional video analysis results of rats' directional movement trajectory are available in Yuan et al. (2014). These observations together with response latency results show that motor skill learning occurred during the first few days and therefore, it could be dissociated from the rest of the associative learning process.

#### Firing Rate Modulation by Directional Choice

In this study, trials with correct first response lever press were used in analysis and within those trials, we mainly focused on the cue-on task period (**Figure 1B**). A total of 220 sessions were recorded from nine rats. We included 190 sessions for analysis and excluded the remaining 30 sessions because of inadequate numbers of trials (less than 20 L-L or 20 R-R trials). Those sessions mainly included the first few sessions of each rat when behavioral accuracy was low and motor skill learning was possibly present. All together for this study, we had 11,060 L-L trials and 10,717 R-R trials from the 190 sessions with 58 L-L trials and 56 R-R trials per session on average. Of the 190 recorded sessions from nine rats, we collected 839 unit records (337 from AGl and 502 from AGm), 4.4 unit records per session on average, ranging from 3 to 6 unit records in the ensemble. Here we consider an isolated unit each day a unit record.

Single unit firing activities of L-L and R-R trials in each session were first inspected by spike rasters and peri-event time histograms (PETHs). Examples of single unit firing rate modulations by L-L and R-R trials are shown in **Figure 4**.

In the 500 ms time window before cue onset, the averaged (over all single units, nine rats, and sessions) single unit firing rate difference between L-L and R-R trials was −0.02 Hz, which was not significantly different from zero (one-sample t-test, p > 0.89). If the same single unit firing rate differences between L-L and R-R trials were evaluated in three cue-on sub-windows (CO1, CO2, and CO3 in **Figure 1B**), they were 2.76, 3.12, and 2.17 Hz, respectively, all of which were significantly greater than zero (one-sample t-test, p < 10−<sup>5</sup> ; **Figure 5A**).

We then evaluated time-resolved (100 ms bins for every 20 ms) single unit firing rates of L-L and R-R trials, and the firing rate difference between the two during the cue-on period (**Figure 5B**). For the pool of single units, the averaged (over all single units, nine rats, and sessions) time-resolved firing rate difference did not emerge from 0 until 400 ms after cue onset (onesample t-test, p < 0.001), and it sustained through the rest of the cue-on window. These results show that firing rate modulation of single neurons was prominent in motor cortical areas during the cue-on period.

To study how firing rate modulation at a population level varied as learning progressed, we calculated the ensemble mean rate and ensemble mean rate difference between L-L and R-R trials (see Materials and Methods) session by session. The results from using rat B11's data are given in **Figure 5C** as an example where the firing rates of a 1500 ms cue-on window (CO, **Figure 1B**) were used. To summarize results from all seven rats, we normalized session numbers. According to **Figure 5D**, the Z-scored ensemble mean firing rate did not change significantly through the learning process (99% confidence interval of linear regression slope: [–0.48 1.05]). When the learning process was divided into three stages of equal numbers of sessions, the Z-scored ensemble mean rates were −0.21 ± 1.08, 0.16 ± 0.85, and 0.00 ± 1.00 Hz (mean ± STD), which were not significantly different (ANOVA, p > 0.1). The Z-scored ensemble mean rate difference between L-L and R-R trials tended to increase slightly with a slope of 0.59 (**Figure 5D**), but its 99% confidence interval was [−0.16 1.34] indicating it was not significantly different from zero. When calculated in the three learning stages, the Z-scored ensemble mean rate differences were −0.16 ± 1.09, −0.19 ± 0.83, and 0.32 ± 0.96 Hz, respectively, showing higher rate differences between L-L and R-R trials in the last stage compared with the previous two stages (ANOVA, p < 0.05). But **Figure 5D** also shows that some early session had large rate differences. When linear regression was performed for individual rats, the slope was again not significantly different from zero (t-test, p > 0.15). To summarize, although ensemble mean rate difference tended to become larger near the end of the recorded learning process, the trend was not strongly observed.

The BC data window (**Figure 1B**) was analyzed in a similar way to provide a control. The ensemble mean firing rate leveled over sessions (slope of Z-scored rates: 0.04, 99% confidence interval of slope: [−0.73 0.81]; Z-scored rates in the three stages were: −0.16±1.06, 0.21 ± 0.77, −0.09±1.08, ANOVA p > 0.1). The ensemble mean rate difference was relatively stable over sessions as well (slope of Z-scored rate difference: −0.40, 99% interval: [−1.17 0.36]; Z-scored rate difference in the three stages were: 0.17 ± 1.01, −0.04 ± 0.92, and −0.09 ± 1.01, ANOVA p > 0.4).

Additionally, the standard deviation of firing rate during the CO data window remained relatively stable across learning sessions. The linear regression slope of mean standard deviation for L-L trials against normalized session number was 0.06, which was not significantly different from zero given that the 99% confidence interval of the slope was [−0.70 0.82] (**Figure 5D**, red dotted line). Similarly for R-R trials, the linear regression slope was 0.04 and its 99% confidence interval was [−0.72 0.80] (**Figure 5D**, green dotted line).

To summarize, the ensemble mean firing rate of all simultaneously recorded single units over all trials did not change significantly over sessions, and the ensemble mean firing rate difference between L-L and R-R trials did not show a clear trend of change either. These findings suggest that ensemble mean rate based measures are not adequate to explain improved behavioral learning of the seven rats under study.

#### Spatiotemporal Firing Pattern Analysis by SVMs

As learning related behavioral adaptation could not be wellexplained by ensemble mean firing rate or ensemble rate difference between L-L and R-R trials in cue-on period, we then inspected the neural data with increased spatiotemporal resolution using SVMs. In the following, we first examined how data preparation and SVM parameters may affect SVM model performance.

#### Ensemble vs. Single Units

Ensemble spike count vectors were formed by concatenating spike counts (CO window, 500 ms bins) of simultaneously recorded single units. For each session, SVMs were trained and tested using ensemble vectors [vector dimension was 3× (number of single neurons)], and classification performance was characterized by classification accuracy on the test data. On the other hand, SVMs were built and tested using spike count vectors of single units (three dimensional vectors). When comparing the best classification performance using single unit data with that using ensemble data (**Figure 6A**), the ensemble approach outperformed the best single unit approach in 61.6% (117/190) of tested sessions. The mean single trial decoding accuracy among all sessions when using ensemble approach was 76.2%, which was higher than the 74.2% accuracy of the best single units (paired-sample t-test, p < 10−<sup>5</sup> ).

Additionally, we examined the impact of the ensemble size on decoding accuracy. Each of the nine rats had at least 3 units per session. Specifically, one rat had 3 units per session, four rats had 4 units per session, three rats had 5 units per session, and one rat had 6 units per session. As shown in **Figure 6A** insert, decoding accuracy increased when larger ensembles were used, but the speed of increase in decoding accuracy by using larger ensembles slowed down as ensemble size increased.

#### Multiple vs. Single Time Bins

To explore SVM classification performance over time, the cueon period was divided into non-overlapping 100 ms time bins and SVMs were trained using data of spike counts in a single time bin from all simultaneously recorded units. As shown in **Figure 6B**, the classification accuracy (averaged over all sessions)

gradually increased after cue onset and then leveled off at around 60% (still above chance level of 50.02% accuracy when training samples from both classes were randomly shuffled, one-sample t-test, p < 10−<sup>5</sup> ). This is consistent with our previous observation of sustained firing rate modulation between L-L and R-R trials during cue-on period (**Figure 5B**). As a comparison, when spike counts in 15 bins (100 ms bin width) together (CO window) were used in SVM model for classification, the average decoding accuracy was 73.01% over all sessions, which was significantly higher than the decoding accuracy when single time bins were used (paired-sample t-test, p < 10−<sup>5</sup> ). Therefore, temporal firing patterns or spike counts in multiple consecutive time bins were expected to benefit SVM neural decoding.

## Size of Time Bin

Then we tested how the size of a time bin may affect SVM classification. The same 1500 ms (CO window) neural ensemble data was used but spikes were counted in non-overlap time bins of different sizes, ranging from 100 to 750ms. Best classification performance was obtained using 500 ms bins with a 76.18% decoding accuracy (**Figure 6C**). Larger time bins (e.g., 750 ms) resulted in slightly lower classification accuracy (76.01% accuracy; pairedsample t-test, p < 0.01) probably due to loss of temporal resolution. However, higher temporal resolutions did not help improve classification accuracy either (paired-sample t-test, p < 0.01). Given the above discussion, we used spike counts in 500 ms bins for analyses hereafter unless otherwise specified.

The histogram of classification performance in single sessions is shown in **Figure 6D**, where spike counts in 500 ms non-overlapping bins during the 1500 ms CO window of neural ensembles were used for decoding. The mean classification accuracy tested with novel single trial data was 76.18% for all sessions from all rats.

To gain additional insight into the firing patterns during the CO window, SVM decoding analysis was also performed between correct and error trials. Two classes of error trials were considered here: R-L trials stand for those left side choices in response to right side cues, and similarly we define L-R trials. Only 11 out of all 190 sessions had at least 20 trials per class for L-L vs. R-L analysis, and 18 sessions were included in R-R vs. L-R analysis. The average decoding accuracy for L-L vs. R-L trials across the 11 sessions was 53.74% which was slightly but significantly higher than chance (t-test, p < 0.03), and the accuracy for R-R vs. L-R decoding was 50.79% which was not significantly different from chance (t-test, p > 0.60). Specifically in L-L vs. R-L analysis, decoding accuracy was higher than chance in 7 out of the 11 sessions (mean accuracy was 57.05%), lower than chance in one session (44.86% accuracy), and not different from chance in the remaining three sessions (t-test, α = 0.001). For the 18 sessions included for R-R vs. L-R analysis, these numbers were six sessions (58.58% accuracy), six sessions (44.38% accuracy), and six sessions. Alongside these decoding results, it's also worth noticing rat behavioral accuracy data. The average behavioral accuracy was 69.1% for the 11 sessions used for L-L vs. R-L decoding, and the behavioral accuracy for the 18 sessions used for R-R vs. L-R decoding was 59.9%. To summarize, when the same directional choice was made in both correct and error trials (e.g., left side choices in both L-L and R-L trials), the neural patterns associated with the two types of trials were largely similar, but there still seemed to be some difference between the two. When the rat was less clear about correct vs. wrong choices (low behavioral accuracy), the neural activities were more similar for correct (R-R) and wrong (L-R) trials. However, this analysis is not conclusive due to limited data available (only 28 out of all 190 sessions were eligible for this analysis).

#### Adaptation of Spatiotemporal Firing Patterns with Learning

Before presenting evidence on neural adaptation as learning took place, we first illustrate how SVMs can be used for this purpose. **Figure 7A** is an example of how SVM classification took place to separate L-L and R-R trials where in the figure, we showed the first two principal components of the original spatiotemporal neural ensemble data. As shown, training data samples of the two classes formed distinct clusters and the SVM created an optimal separating line properly. This classifier was then used to predict rat response of left or right side lever press given a novel single trial neural data sample from the test data set (**Figure 7B**). The classification accuracy on the test set and the averaged decision function values (see Materials and Methods) of test samples from both classes could be calculated.

**Figure 7C** shows histograms of decision function values of test neural data samples from the 100 runs of randomly selected test samples in two sessions (session 9 and 21) from rat B11 as an example. Decision function values from the two sessions were significantly different between the two classes (ANOVA, p < 10−<sup>5</sup> ). But the distance between the mean decision function values of the two classes (0.40 vs. 1.14) was larger in the later session when SVM classification accuracy was also higher (53.08% vs. 64.30%).

As classification accuracy increased over learning sessions, we also see an increase in the distance measurements of decision function values of the two classes provided by SVM (**Figure 7D**, rat B11 as an example). Among the seven rats, linear regression of Z-scored classification accuracy (CO data window, 500 ms bins) against normalized session number had a positive slope of 1.59 (**Figure 7E**), with the 99% confidence interval at [0.91 2.26]. If we divide learning sessions into three stages of equal length, the average of the Z-scored classification accuracy gradually increased over the three stages at −0.52, −0.19, and 0.48 (1st vs. 3rd stage: ANOVA, p < 0.001; **Figure 7E**). The Z-scored distance measurement between decision function values of the two classes had a positive regression slope of 1.32 (99% confidence interval at [0.62 2.02]), and increased significantly over the three stages as well to reach their respective Z-scored distance measurement of −0.44, −0.15, and 0.40 (1st vs. 3rd stage: ANOVA, p < 0.001; **Figure 7E**). These results suggest enhanced discriminability in spatiotemporal neural activity patterns between L-L and R-R trials as learning progressed.

After examining neural activities in the cue-on period in relation to rat's behavioral learning improvement, we attempted to gain additional insight by investigating neural activity patterns during the response lever press period as a control. A 500 ms time window (LP window, from −100 to 400 ms) around response lever press was used, and spike counts in 100 ms non-overlapping bins were used to build SVMs to decode L-L and R-R trial lever presses. For this time window, Z-scored classification accuracy exhibited a weak rising trend (**Figure 7F**, LP window), with the slope of a linear regression at 0.55, which was not significantly different from 0 since the 99% confidence interval of the slope was [−0.23 1.34].

To compare with those results using the LP data window, we repeated the analysis for the three cue-on period windows (**Figure 7F**). For CO1, from 0.5 to 1.0 s after cue onset while directional movement was being performed, improvement of Zscored classification accuracy was significant (slope: 0.98; 99% confidence interval of slope: [0.25 1.71]). For CO2, from 1.0 to 1.5 s after cue onset when directional movements were mostly completed, improvement of Z-scored classification accuracy was significant as well (slope: 1.48; 99% confidence interval of slope: [0.79 2.17]). An improvement of Z-scored classification accuracy (slope: 1.12; 99% confidence interval of slope: [0.40 1.84]) was also observed in CO3, from 1.5 to 2.0 s after cue onset which was right before extension of response levers. When the regression analysis was carried out on individual rats, regression slopes for cue-on period data windows were significantly greater than zero (t-test; CO1, p < 0.01; CO2, p < 0.005; CO3, p < 0.05), but not significantly different from zero for LP (t-test, p > 0.39). When classification accuracies were averaged for each of the three equallength learning stages, Z-scored classification accuracy increased gradually over stages for each of the three cue-on period windows (CO1: −0.76, −0.37, and 1.13; CO2: −0.86, −0.24, and 1.10; CO3: −1.05, 0.11, and 0.94; ANOVA, 1st vs. 3rd stage, p < 0.05), but remained leveled during the last two stages for LP window (−1.15, 0.67, and 0.48; ANOVA, 2nd vs. 3rd stage, p > 0.9). **Figure 7G** illustrates the Z-scored classification accuracies in the three stages for the four data windows, and again consistent increment over the three stages was found in cue-on data windows but not in the LP window. Taken together, enhanced discriminability of neural activity patterns over the entire learning process was mainly found during the cue-on period, but not the LP period when rats actually pressed levers.

To further validate the results from SVM based decoding analyses presented above, we used linear discriminant analysis (LDA) as a second classification method. Similar results from LDA classifiers were obtained as those reported in **Figures 7E–G**.

Regression analysis was also performed between classification accuracy (CO data window, 500 ms bins) and the percentage of trials with correct directional choice in single sessions. Results of the seven rats were plotted in **Figure 8A** individually. The linear regression slope was significantly greater than zero (t-test, p < 0.05). Thus, decoding of directional choice using neural activity did improve as rats made progress on the learning task.

To show that spatiotemporal patterns indeed facilitated the observed improvement in directional choice decoding, the mean firing rate of the neural ensemble over the whole CO data window (1500 ms bin, 1-D data samples) was used for classification as a comparison. Regression slope of classification accuracy against the percentage of correct choice trials (i.e., L-L and R-R trials) when using the 1-D data samples is −0.0071 (**Figure 8B**, black; n = 7), which is not significantly different from zero (99% confidence interval at [−0.0221 0.0078]). When spatiotemporal patterns (CO data window, 500 ms bins) were used for classification, the regression slope is significantly steeper (ANCOVA, p < 0.05), which is 0.0380 with 99% confidence interval at [0.0147 0.0612]. Therefore, neural adaptation associated with directional choice learning is better described by spatiotemporal activity patterns than a low resolution neural activity representation.

# Discussion

Seven out of nine rats successfully learned to perform a directional choice task from a naïve state. Using trial-and-error, they were able to associate a light cue with a same side lever pressing. Based on rat behavioral data, we observed that rat movement trajectory and the act of lever press became stereotyped within the first few days and therefore, the motor skill learning factor could be excluded from our analysis of associative learning. In this study, we focused on analyzing neural data from the seven

FIGURE 7 | SVM decoding over learning sessions. The CO data window with spike counts in 500 ms time bins was used in all results except those in (F) and (G) where 100 ms bins were used. (A) An illustration of training data samples, support vectors and separating plane (black line) in a 2-D PCA projected space. Data of rat B11 in session 21 were used as an example. (B) The original single trial data samples used to generate trial-averaged training data and single-trial test data samples plotted in the same space as in (A). (C) Histograms of decision function values of test data set for sessions 9 and 21 of rat B11. (D) An example of classification performance over sessions using data of rat B11. Upper panel: Classification accuracy tended to

increase with learning. Lower panel: the distance between L-L and R-R data sets in the SVM kernel space increased with learning. (E) Classification accuracy and distance between the two classes in SVM kernel space increased through the course of learning (seven rats). (F) The increased decoding accuracy was significant in the three cue-on period data windows (CO1, CO2, and CO3), but not in LP window around response lever press. (G) Neural activity patterns of response lever press (LP) showed different dynamics in terms of classification accuracies (Z-scored) in three stages of the learning process compared with those in cue-on data windows (CO1, CO2, and CO3).

rats during the cue-on period when they made directional choice decisions. Our results showed that the ensemble mean firing rate over all L-L and R-R trials appeared level over learning sessions. The ensemble mean rate difference between L-L and R-R trials did not show a strong trend of change as task learning progressed either (**Figure 5**). However, when using SVMs to decode directional choice from spatiotemporal neural activity patterns, there was a clear upward trend of SVM decoding accuracy over learning sessions. Correspondingly, there was a clear upward trend in discriminability of neural patterns between left side and right side choices (**Figure 7**). These findings suggest that neural adaptation in rat's motor cortical areas during learning of the directional choice task may lie in the spatiotemporal firing pattern of neural ensembles.

# Adaptation of Spatiotemporal Neural Activity Pattern during Task Learning

SVM classifiers were constructed to discriminate neural activity patterns associated with directional choices. During our analysis using SVM, care was given to ensure compatibility when comparing results over learning sessions. First, the same linear kernel SVM model was used for all analyses. Second, each and every classifier was trained with the same numbers of trial samples and tested with the same numbers of samples in the same session (**Figures 6**–**8**). Third, we reported classification accuracy in each session using an averaged result of 100 independent SVM classifiers with randomly selected training and test data samples. As such, SVM classification performance over sessions as reported in **Figures 7**, **8** should be characteristic of neural activity patterns as they adapted with learning.

In this study, we treated single units recorded from the same electrode in different sessions as independent unit records. Actually, our results reported in this study were based on neural ensembles which consisted of all units recorded simultaneously during one session from the same rat. Therefore, we did not intentionally identify and track same neurons over recording sessions. This approach was used before (Cohen and Nicolelis, 2004) and it is adopted in this study since all our results are based on ensemble neural activity.

Motor skill learning is a possible confound when analyzing neural activity in the cue-on period. However, we observed that rats became accustomed to directional movement and response lever press faster than the associative learning aspect of the task. Rat's directional movement became stereotyped quickly after the first few sessions (**Figure 3D** and Yuan et al., 2014) and the response latency of lever press decreased during the first three sessions (**Figure 3B**). Therefore, initial motor skill learning could not explain neural adaptation along an entire learning process lasted for weeks.

There may be other confounding factors in addition to motor skill learning during the first few learning sessions. Notice that right after a press on the center ready lever, at which time the directional cue was presented, there was a 2 s cue-on period. Rats usually moved from the center position to the location of their chosen response lever waiting for the extension of the response levers at the end of the cue-on period. Conceivably during this 2 s window, a rat's anticipation of lever extension and planning of lever press could possibly induce neural modulation. However, behavioral data showed that rats quickly became accustomed to performing the task routine after the first few days as indicated by their stereotyped movement and stable response latency. Therefore, they could quickly become habituated to the extension of response levers as well as the planning of the routine act of response lever press. From that, those potential confounding factors may be excluded from possible reasons for reported neural adaptation.

Would the reported neural pattern adaptation be explained by the repetition of directional movements? Our previous analysis showed that rat directional movement was mostly completed within CO1 window (Yuan et al., 2014), and there was no obvious or systematic movement during CO2 and CO3 windows (**Figure 1B**). Improved discriminability of neural patterns, however, was observed not only in CO1 but also in CO2 and CO3 windows (**Figure 7F**). Response lever press was another action repeatedly performed by the rats during learning of the task. However, we did not observe significantly improved discrimination of neural patterns in the LP data window (**Figure 7F**). As discussed above, these actions quickly became stereotypical and turned into learned motor skills. Previous primate studies showed that mere repetition of familiar actions did not induce systematic changes in motor cortical neural firing activity (Paz and Vaadia, 2004; Rokni et al., 2007). Taken together, the observed firing pattern adaptation during cue-on period was unlikely to be associated with repeated performance of task related actions.

Reward based stimulus-response association learning is another aspect that may affect neural modulation in the rat frontal areas that we recorded from. Previous studies showed that reward-related action selection is believed to be mediated by the corticostriatal circuitry, linking prefrontal (PFC), premotor, sensorimotor cortices, and the striatum (Balleine et al., 2007). As two important nodes within this circuit of primates, Pasupathy and Miller (2005) found rapid changes in striatum but slow adaptation in PFC during an associative learning task, where the time course of PFC activity had a significantly stronger correlation with the gradual improvement in task performance. In accordance with their findings, the adaptation we observed in rat's frontal areas also correlated with slow improvement of behavioral performance. Given the above considerations, the observed neural adaptation during the cue-on period could be attributed to learning the correct stimulus-response association.

In the rat brain, both AGm and AGl project to basal ganglia (Reep et al., 1987; Cheatwood et al., 2003; Alloway et al., 2009), and both areas receive inputs from basal ganglia through the thalamus (Donoghue and Parham, 1983; Reep et al., 1984). Rat AGm and AGl have connections with a variety of frontal cortical areas as well (Reep et al., 1984, 1990; Hoover and Vertes, 2007). Therefore, these rat motor cortical areas could be in the loop of the reward-related decision making circuit. The neural pattern adaptation in motor cortical ensembles reported here provides neurophysiological evidence for a role of rat motor cortical areas in learning stimulus-response associations, which could be mediated by this neural circuit when rewarded directional choices were learned.

# Ensemble Mean Rate vs. Spatiotemporal Activity Pattern

Despite the well-observed phenomenon of dynamic neural modulation in single neurons, relatively uniform firing rates in cortical ensembles have been reported in different brain areas of both primates and rodents while animals performed different tasks (Hoffman and McNaughton, 2002; Carmena et al., 2003; Costa et al., 2006; Pantoja et al., 2007). But that does not include studies that involve learning tasks except a few. In an associative learning task, Cohen and Nicolelis (2004) reported unchanged rate difference over the recorded neural ensemble during the early days, specifically day 2 and day 3. When monkeys were performing a stimulus-response association learning task, firing rate change in single units of motor cortical areas was reported (Mitz et al., 1991; Chen and Wise, 1995; Brasted and Wise, 2004), but it was less certain if and how firing rate of a neural ensemble would change over learning. Here, we report a relatively stable ensemble mean firing rate and ensemble mean rate difference between left and right side choices over a period of about 20–30 sessions covering the entire time course of associative learning. Our results may suggest a balanced increase and decrease in single unit firing rates, which may have contributed to a stable motor cortical ensemble mean firing rate during associative learning. This result is supportive of "the conservation of firing" principle proposed by Nicolelis and Lebedev (2009).

To gain understanding of neural coding beyond ensemble averaged firing rates, investigations of spatiotemporal activity patterns at a fine resolution have brought up new insight on fundamental neural mechanisms in visual attention (Heinze et al., 1994), odor representation (Laurent et al., 1996; Spors and Grinvald, 2002; Rennaker et al., 2007), auditory processing (Kayser et al., 2009), vibrissa deflection coding (Petersen and Diamond, 2000), contextual encoding (Hyman et al., 2012), sequence learning (Ma et al., 2014), and rule learning (Durstewitz et al., 2010), to name a few. In a reaction time study (Laubach et al., 2000) using a rat model, the overall firing rates of the AGm and AGl ensembles did not change significantly but prediction of trial outcome of either correct or error based on spatiotemporal activity patterns improved over learning sessions. Our results appear along similar lines. However, the two experimental protocols are different in a few aspects. The Laubach et al. (2000) experiment used a single stimulus (vibrotactile or auditory) and a lever release for rats to report their detection of presence of sensory cue. In our experiment, alternative choices were associated with distinct stimuli. Besides, Laubach et al. (2000) compared lever release either instructed by a stimulus (correct) or executed spontaneously without stimulus presence (error), while we compared instructed left side and right side choices (both were correct) under distinct cues. This may help rule out confounding factors such as the occurrence of sensory stimuli and prediction of rewards (Carandini and Churchland, 2013).

In the Cohen and Nicolelis (2004) study, prediction of left and right side movements by M1 neural ensembles improved from the first day to the next 2 days. Unfortunately, results in the remaining 8 days as rats' performance continued to improve until reaching a plateau were not available. Here we monitored neural activity patterns through the entire process of associative learning, and demonstrated improved discriminability of spatiotemporal firing patterns in motor cortical ensembles.

# Rat Motor Cortical Areas and Associative Learning

The rat AGl area has been considered to correspond with primate primary motor (M1) cortex (Donoghue and Wise, 1982; Donoghue and Parham, 1983). On the other hand, the rat AGm area refers to the medial subdivision of the agranular field of the frontal cortex which differs from the lateral subdivision (AGl) on cytoarchitectonic grounds (Donoghue and Wise, 1982). Other terms referring to this area used in literature include medial precentral area (PrCm, Krettek and Price, 1977), frontal cortical area 2 (Fr2, Zilles, 1985), and secondary motor area (M2, Paxinos and Watson, 2005; MOs, Swanson, 1998). Leaving the inconsistent nomenclature aside, rat AGm was proposed to be homologous to premotor cortex, supplementary motor area, and frontal and supplementary eye fields in primates (Donoghue and Wise, 1982; Reep et al., 1987; Van Eden et al., 1992; Condé et al., 1995; Sul et al., 2011). However, a clear rat AGm homology in primates has yet to be proved convincingly. Nonetheless, previous neuropsychological studies showed that lesions of rat AGm impaired both the retrieval (Passingham et al., 1988) and the acquisition (Winocur and Eskes, 1998) of visuomotor conditioning, which suggest a role for AGm in stimulus-response associative learning. In line with these reports, we observed firing pattern adaptation during associative learning within the motor cortical neural ensembles, which consisted of AGm neurons and AGl neurons.

It is worth mentioning that in most of the primate studies on associative learning, the animals usually had acquired certain stimulus-response association through long and extensive training. And then, animals would learn a novel pairing (Chen and Wise, 1995; Brasted and Wise, 2004) and/or the reversed pairing (Pasupathy and Miller, 2005; Histed et al., 2009) during one complete recording session. Sometimes the animals would learn a variant of the trained sensorimotor tasks (Mitz et al., 1991; Li et al., 2001; Genovesio et al., 2014), which also could be completed within one session. In these cases, the complete time course of learning could take place in tens of minutes or a single recording session. This may be too soon to result in long-lasting synaptic changes, as suggested by Histed et al. (2009). In our experiment, it took rats several weeks to master the directional

# References


choice task. This may allow substantial neural adaptation to take place, possibly through changes at the synaptic level.

Synaptic plasticity has long been hypothesized for being an important neurochemical foundation of learning and memory (Malenka and Bear, 2004; Gilson et al., 2010), and its necessity has been well-supported (Martin et al., 2000). Rat motor cortex is highly capable of functional and structural changes even in adulthood. Reorganization of motor maps has been observed in various experiments (Sanes et al., 1990, 1992; Lee et al., 2003), including animals learning a motor skill (Nudo et al., 1996; Kleim et al., 1998, 2004). Cortical synaptogenesis has been reported during motor training (Jones et al., 1999; Kleim et al., 2004). And recent studies demonstrated learning-induced dendritic spine changes in rodents performing motor tasks (Xu et al., 2009; Yang et al., 2009; Wang et al., 2011). While these changes were related to learning of certain motor skill, whether learning stimulusresponse association would induce such changes in rat motor areas is unclear. Given the lengthiness of the task, rats in our experiment may have a chance to experience synaptic modification during the learning process that lasted several weeks. Consequently, an enhanced spatiotemporal neural representation may become increasingly predictable of directional choice.

# Acknowledgments

This work was supported in part by the National Science Foundation under Grants ECS-0702057 and ECS-1002391.


Gilson, M., Burkitt, A., and Van Hemmen, J. L. (2010). STDP in recurrent neuronal networks. Front. Comput. Neurosci. 4:23. doi: 10.3389/fncom.2010.00023


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Mao, Yuan and Si. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Capture of fixation by rotational flow; a deterministic hypothesis regarding scaling and stochasticity in fixational eye movements

#### *Nicholas M. Wilkinson1 \* and Giorgio Metta1,2*

*<sup>1</sup> iCub Facility, Fondazione Istituto Italiano di Tecnologia, Genova, Italy*

*<sup>2</sup> Centre for Robotics and Neural Systems, School of Computing and Mathematics, University of Plymouth, Plymouth, UK*

#### *Edited by:*

*Ruben Moreno-Bote, Foundation Sant Joan de Deu, Spain*

#### *Reviewed by:*

*Susana Martinez-Conde, Barrow Neurological Institute, USA Zachary P. Kilpatrick, University of Houston, USA*

#### *\*Correspondence:*

*Nicholas M. Wilkinson, iCub Facility, Fondazione Istituto Italiano di Tecnologia, via Morego, 30 Genova, Italy e-mail: nicholas.wilkinson@iit.it*

Visual scan paths exhibit complex, stochastic dynamics. Even during visual fixation, the eye is in constant motion. Fixational drift and tremor are thought to reflect fluctuations in the *persistent neural activity* of neural integrators in the oculomotor brainstem, which integrate sequences of transient saccadic velocity signals into a short term memory of eye position. Despite intensive research and much progress, the precise mechanisms by which oculomotor posture is maintained remain elusive. Drift exhibits a stochastic statistical profile which has been modeled using random walk formalisms. Tremor is widely dismissed as noise. Here we focus on the dynamical profile of fixational tremor, and argue that tremor may be a signal which usefully reflects the workings of oculomotor postural control. We identify signatures reminiscent of a certain flavor of transient neurodynamics; toric traveling waves which rotate around a central phase singularity. Spiral waves play an organizational role in dynamical systems at many scales throughout nature, though their potential functional role in brain activity remains a matter of educated speculation. Spiral waves have a repertoire of functionally interesting dynamical properties, including persistence, which suggest that they could in theory contribute to persistent neural activity in the oculomotor postural control system. Whilst speculative, the *singularity hypothesis* of oculomotor postural control implies testable predictions, and could provide the beginnings of an integrated dynamical framework for eye movements across scales.

**Keywords: fixational eye movement, tremor, traveling waves, spiral wave, phase singularity, Lévy walk, persistent neural activity, neural integrator**

#### **1. INTRODUCTION**

During fixation the eye is not still. Three main classes of *fixational eye movement* (henceforth FEM) have been identified (Martinez-Conde et al., 2004). Microsaccades are very fast movements which occur relatively infrequently. Drift is a slow, meandering component which occupies most of fixation time. Tremor is a fast, low amplitude aperiodic oscillation imposed on drift. Microsaccades are in many ways much like saccades on a tiny scale (Ko et al., 2010; Kagan and Hafed, 2013; Martinez-Conde et al., 2013; Otero-Millan et al., 2013; Poletti et al., 2013), though they may also be linked to the drift component (Engbert and Mergenthaler, 2006; Engbert et al., 2011). FEM have classically been thought to counteract sensory adaptation. Recent evidence suggests that FEM play a more sophisticated role, optimizing visual flow for the response properties of retinal ganglion cells (Rucci et al., 2007; Kuang et al., 2012) and relocating the highest resolution parts of the retina with great precision (Ko et al., 2010; Poletti et al., 2013). Some theories suggest that FEM perform an active perceptual palpitation of the visual scene which is fundamental to vision (Ahissar and Arieli, 2001, 2012; O'Regan and Noë, 2001). Recently, very high resolution eye movement data based on tracking tiny movements of ocular vein structure in three dimensions has revealed more structure to FEM than had previously been suspected (Li and Zhang, 2012; Zhang and Li, 2012). These studies reported microsaccades which were not straight and ballistic (as previously thought), but curving, and even bent and jerky. Relatively little detailed information was given, but it was reported that the drift-tremor combination took a complex, curling trajectory. These high resolution data may enable new insight into the underlying generative mechanisms of fixational eye movements. Oculomotor postural control is mediated by brainstem circuits (Aksay et al., 2000, 2007; Sparks, 2002) and is strongly associated with *persistent neural activity* (Major and Tank, 2004), which plays the role of *integrating* transient stimulation from superior colliculus reflecting saccadic velocity commands into persistent activity encoding the new eye position. The neuroanatomy and functional circuitry of oculomotor postural control has been intensively studied (e.g., Aksay et al., 2000, 2001, 2003; Miri et al., 2011a,b; Fisher et al., 2013), but the precise mechanisms underlying drift and tremor remain elusive.

Rotational waveforms (aka spiral waves, vortices, tori) are a commonplace, universal dynamical form which play an organizing role in dynamical systems at all scales, from galaxies to weather to evolution to organisms to organs to cells to photons (Toomre, 1969; Da-sheng, 1980; Boerlijst and Hogeweg, 1991; Gray and Jalife, 1996; Winfree, 2001; Molina-Terriza et al., 2007; Schecter et al., 2008; Taniguchi et al., 2013). This generality led Winfree (2001) to suggest toroidal temporal structure as a fundamental aspect of biological time, a notion for which the reference provides many empirical examples. Spiral waves are a canonical mode of pattern formation in dissipative systems operating far from equilibrium (Kuramoto and Koga, 1981; Cross and Hohenberg, 1993). The brain, by necessity, is one such system (e.g., Kelso, 1995; Ermentrout, 1998). Thus if toroidal waveforms (termed *spiral waves* in two dimensions, and *scroll waves* in three dimensions) do not play an organizing role in normal neurodynamics, then the brain must be considered something of an exception to the rule, which would require explanation. The suspected role of spiral waves in some pathological scenarios such as epilepsy (Milton, 2012) and cardiac fibrillation (Gray et al., 1998) suggests that the nervous system possesses mechanisms for actively suppressing turbulence and spatiotemporal chaos (e.g., Schiff et al., 1994), but observations of spiral waves in non-pathological settings (Jung et al., 1998; Huang et al., 2004, 2010) clarify that such suppression is not complete or universal. Indeed, an active field of study in cardiac defibrillation is the suppression of spatiotemporal chaos and turbulent neural activity by the seeding of spiral waves (e.g., Zhang et al., 2002; Xiao-Ping et al., 2011).

The current contribution hypothesizes a connection between quasi-persistent spiral neurodynamics and *persistent neural activity* in the context of oculomotor postural control (Major and Tank, 2004). Sections 2 and 3 respectively introduce the literature on spiral waves and fixational eye movements. Section 4 details our motivations in proposing the *singularity hypothesis* of postural memory. Our purpose is not an exhaustive review, nor to convince the reader that our hypothesis is necessarily correct, but a targeted presentation of empirical evidence and functional arguments which render the singularity hypothesis interesting, plausible and worth testing. Section 5 offers some concluding remarks. Predictions are presented in boxes in the main text.

#### **2. TRANSIENT NEURODYNAMICS AND SPIRAL WAVES**

The classical focus on attractor networks in systems neuroscience (see for review Amit, 1992) is increasingly being enriched by a modern synthesis which also stresses the importance of selforganization and transient neural dynamics (Rabinovich et al., 2001; Maass et al., 2002; Seliger et al., 2003; Durstewitz and Deco, 2008; Friston et al., 2012; Milton, 2012), fractality in physiology (Goldberger and West, 1987; West et al., 1994; Werner, 2010; West, 2010), self-organizing criticality (Bak et al., 1987; Bak, 1996; Jung et al., 1998), chaotic itinerancy (Tsuda, 1991, 2001; Kaneko and Tsuda, 2003) and dynamic pattern formation in non-equilibrium dissipative systems (Cross and Hohenberg, 1993).

Neural spiral waves are an intriguing class of quasi-persistent transient neurodynamics, whose functional potential in brain activity remains an open question. They have received extensive theoretical attention in terms of their abstract properties in networks (e.g., Coombes, 2005; Kilpatrick and Bressloff, 2010b; Ma et al., 2012b), but surprisingly little attention in terms of concrete cases linking their dynamics to perception and behavior. We have conducted preliminary modeling studies employing spiral waves for visual salience mapping (Wilkinson and Metta, 2011; Wilkinson et al., 2011), and spiral neurodynamics have linked to visual geometric hallucination (Bressloff et al., 2001; Kilpatrick and Ermentrout, 2012a,b; Froese et al., 2013). At the motor end, Heitmann, Breakspear and colleagues have developed physiologically explanatory models showing how traveling waves (including spirals) can encode motor trajectories (Heitmann, 2013).

#### *2.0.1. Spiral waves in nature, biology, and the brain*

The multiscale ubiquity of spiral waves in nature and biology (Toomre, 1969; Lechleiter et al., 1991; Winfree, 2001), and their interesting dynamical properties (Boerlijst and Hogeweg, 1991; Biktashev and Holden, 1993, 1995; Langham and Barkley, 2013), have motivated many physical, chemical, and mathematical studies. Arthur Winfree pioneered computational and empirical investigations of toroidal dynamics in chemical and biological systems (Winfree, 1967, 1972). Many biological dynamics exhibit toroidal form (Winfree, 2001). The modern understanding of pathological heart fibrillation (and de-fibrillation intervention) is perhaps the most prominent medical application of this work (e.g., Gray et al., 1998; Gray and Chattipakorn, 2005), though cellular calcium dynamics is another important example (Lechleiter et al., 1991). Spirals are reentrant waves which circle around a central rotor known as a *phase singularity* (Winfree, 1991); a point of maximally uncertain phase, surrounded by points of all phases. The central rotor of a whirlpool or tornado provides a physical example in three dimensions.

Propagating calcium waves in astrocyte networks are thought to play an important role in regulating brain activity (Cornell-Bell and Finkbeiner, 1991; Finkbeiner, 1992). Jung et al. (1998) observed that *Ca*2<sup>+</sup> spiral waves exhibiting scale-free distributions suggestive of self-organizing criticality (Bak et al., 1987) are characteristic of healthy function, whilst epileptic events are characterized by the breakdown of this scaling. In neural tissue, traveling waves have been observed widely in various species in both sensory and motor cortices (see for review Wu et al., 2008; Sato et al., 2012) via voltage sensitive dye imaging ("VSDI"). VSDI is an invasive optical imaging method which enables measurement of subthreshold changes in membrane potential with high spatiotemporal resolution (Grinvald and Hildesheim, 2004). Spiral dynamics are commonplace in the dynamics of simulated excitable media including networks of model neurons (Milton et al., 1993; Winfree, 2001; Chun-Ni et al., 2010; Yu et al., 2010; Ma et al., 2012a), and have been observed in mammalian (Huang et al., 2004, 2010) and reptilian (Prechtl et al., 1997) cortex. Movies of cortical spiral waves in the VSDI signal (from Huang et al., 2010) can be found http://www9.georgetown.edu/faculty/wuj/propagationwave.html.

It has been suggested that spiral waves may play an organizing role in neural field interactions (Wu et al., 2008; Freeman, 2009; Huang et al., 2010). Short-lived spiral waves are frequently observed in the healthy case (Huang et al., 2010), but the growth of spiral wave formations of large duration and extent has been linked to pathological conditions including heart fibrillation (Gray et al., 1998) and epileptic seizure (Milton and Jung, 2003; Viventi et al., 2011; Milton, 2012; Stacey, 2012). This is suggestive that spiral waves are a part of normal function, whether constitutive or epiphenomenal, but that their (potentially useful) tendency to enslave surrounding dynamics (e.g., Savill et al., 1997; Yang and Yang, 2007; Huang et al., 2010) has to be carefully controlled.

#### *2.0.2. Dynamical behavior of spiral waves*

The dynamical behavior of spiral waves can be complex and is the subject of extensive research. A useful introduction with video visualizations is given at Björn Sandstede's website hosted by the Department of Applied Mathematics at Brown University, USA, http://www.dam.brown.edu/people/sandsted/research. php?project=spirals. At the risk of oversimplifying, the behavior of spiral singularities exhibits three basic components. Firstly, the rotational orbit of the whole spiral wave will exhibit a characteristic frequency and phase, and is reflected in a small, "on the spot" circular rotation of the phase singularity at the spiral tip. The current hypothesis proposes that this component may correspond to the low frequency peak observed in tremor statistics (Spauschus et al., 1999; Greschner et al., 2002), and could explain new high resolution observations of curling trajectories of drift and tremor (Li and Zhang, 2012; Zhang and Li, 2012).

This trajectory may be perturbed in various ways to take on a locally more complex, globally drifting form. This is known as spiral *drift*, and occurs in response to various forms of symmetry breaking perturbations/gradients in the external milieu (Biktashev and Holden, 1995; Wulff, 1996; Sandstede et al., 1999; Biktashev, 2007). Figures 1–11 in the Scholarpedia article Biktashev (2007) (http://www.scholarpedia.org/article/ Drift\_of\_spiral\_waves), display images and animated movies of the trajectory of spiral singularities under various forms of symmetry breaking. Note the basic curling trajectory, whose period is equal to that of the wave's orbit. This spiral drift in response to symmetry breaking perturbations in the excitability of the medium is the neural correlate we hypothesize for the well known slow component of fixational drift (Martinez-Conde et al., 2004; Rolfs, 2009).

In addition to the above relatively slow components, fast, aperiodic oscillatory modulations of the basic curling trajectory can result from instabilities at the phase singularity (Winfree, 1991). The singularity is the point at which all surrounding signals cancel exactly, and so small fluctuations in the surround cause this point of balance to jitter unpredictably. Gray et al. (1998); Bray et al. (2001) tracked the spacetime trajectory of phase singularities in cardiac fibrillation data. **Figure 1**, from Bray et al. (2001) depicts the evolution of a real cardiac phase singularity (white tube inside black mesh) in detail over one cycle of the carrying spiral wave. **Figure 2**, also from Bray et al. (2001) graphs longer trajectories of the singularities of four interacting spirals. Note the fast (80–90 Hz) aperiodic oscillation superimposed on the basic curling trajectory, much faster than the period of the carrier wave. Gray et al. (1998) reported similar spiral meander during cardiac fibrillation. Though here in cardiac tissue, this instability at the singularity is a universal feature of spiral waves (Winfree, 1991). The current proposal suggests this instability as the source of the well known high frequency component of fixational tremor (Martinez-Conde et al., 2004; Rolfs, 2009).

#### *2.0.3. The functional role of neural traveling waves*

Traveling waves are routinely observed throughout the brain (Wu et al., 2008), and evidence is increasingly suggesting that they play a functional role (Modolo et al., 2011; Sato et al., 2012; Bahramisharif et al., 2013). Heitmann, Breakspear and colleagues have produced a series of physiologically explanatory and plausible models showing how traveling waves can encode motor trajectories read out by dendritic spatial filters (Breakspear et al., 2010; Heitmann et al., 2012, 2013). These are particularly interesting in the current context. In these models, traveling waves encode motor patterns defining *movement*, whilst synchrony constitutes the *resting state*. The current model, in which spiral waves encode for the active holding of posture, sits well in this framework, because spirals, unlike other traveling waves, *have and (almost) hold a location* in a specific sense (Biktasheva and Biktashev, 2003; Langham and Barkley, 2013). This makes them interesting for the kind of active almost-stillness characterizing postural control.

Spiral wave activity has been observed in the VSDI signal, which primarily reflects the field dynamics of sub-threshold membrane potentials (Grinvald and Hildesheim, 2004). These waves can keep cells in a depolarized "ready" state for input, or indeed polarize cells to effectively ignore input (Bahramisharif et al., 2013). This implements a form of spatiotemporally structured *gain control*, widely agreed to be a fundamental aspect of nervous function (Hillyard et al., 1998; Salinas and Thier, 2000; Salinas and Sejnowski, 2001; Rothman et al., 2009; Olsen et al., 2012). Gain fields have been associated with attentional selectivity at both the sensory and motor end (Aston-Jones and Cohen, 2005; Saalmann and Kastner, 2009; Sara and Bouret, 2012).

On this view, fast, aperiodic spiral meander depolarizes a point locus of local cells in the *gamma* band (peaking around 80–90 Hz)

**FIGURE 1 | The evolution of a spiral wave over one rotational orbit.** Space is represented in the horizontal axes, and time (in milliseconds) on the vertical axis. The black mesh encloses a thresholded area of reduced variance (i.e., low amplitude) at the spiral center, as observed in cortex by Huang et al. (2004, 2010). The white tube within the black mesh tracks the evolution of the phase singularity at the spiral core. Note the fast (80–90 Hz) oscillation of the singularity, which we hypothesize underlies the fast component of fixational tremor. Reproduced from Figure 8B in Bray et al. (2001), copyright John Wiley and Sons Publishing 2001.

in a quasi-phaseless manner. The region outside the spiral center is polarized and depolarized periodically by the spin of the spiral arms, on a slower (5–25 Hz) scale dependent on the period of the spiral orbit and the number of spiral arms. Examining the relationship between local field potential and spike rates in the temporal cortex, Zanos et al. (2012) found two populations of cells with just these response characteristics. One population responded at high frequencies in a phase invariant manner, the other at lower frequencies in a phase dependent manner. The (quasi)persistent, self-generating character of spiral waves is particularly interesting in the context of persistent neural responses to transient stimuli. Huang et al. (2010) suggests that spiral waves in visual cortex may be involved in maintaining persistent activity from transient stimuli in the sensory context. A video of their minimal computational model, in which a persistent spiral wave is seeded by transient input, is http://www.jneurosci.org/content/ suppl/2004/11/03/24.44.9897.DC1/model-\_spiraldrift.mpg. The current hypothesis extends this idea to the context of persistent activity in oculomotor postural control (Aksay et al., 2001; Major and Tank, 2004).

#### **3. FIXATIONAL EYE MOVEMENTS**

Fixational eye movements can be quite different between species. Martinez-Conde and Macknik (2008) review comparative studies of FEM in different species, concluding that tremor appears to be the most phylogenetically conserved and fundamental component, consistent with a basic role for spiral wave dynamics in the generative process of FEM. Drift is also widespread, whilst microsaccades appear linked to the existence of foveated ocular architecture. Microsaccades are the most intensively researched component of FEM in humans. These fast relocations of the fixation point appear to play a similar role and manifest similar neural correlates as saccades more generally (Ko et al., 2010; Hafed and Krauzlis, 2012; Kagan and Hafed, 2013; Martinez-Conde et al., 2013; Otero-Millan et al., 2013; Poletti et al., 2013), but also show relations to drift (Engbert and Mergenthaler, 2006; Chen and Hafed, 2013). Microsaccades are relatively infrequent, occuring up to three times per second at most and usually less frequently, in an irregular but individually characteristic fashion (Engbert and Mergenthaler, 2006).

Most of fixation time (*>*90%) is occupied by a slow *drift* of fixation (Martinez-Conde et al., 2004; Martinez-Conde, 2006; Rolfs, 2009), as depicted in **Figures 3**, **4**. Upon this is superimposed a fast (peaked around 80–90 Hz), low amplitude (approx. one photoreceptor), aperiodic oscillation termed *tremor*. Tremor is usually within the noise range of the recording equipment (Martinez-Conde et al., 2004). As a result, less is known about tremor than other components, and tremor is not resolved in many FEM studies of drift and microsaccades.

#### **3.1. DYNAMICAL CHARACTERISTICS OF FIXATIONAL EYE MOVEMENTS**

#### *3.1.1. Random walk modeling of FEM*

On the basis of early studies (Cornsweet, 1956; Matin et al., 1970; Findlay, 1971), FEM have been widely held to exhibit the 1*/f* spectrum of classical Brownian noise. Random walk analysis examines the *mean squared displacement* of a diffusing "particle" (in this case the point of fixation) relative to time. In ideal

**1 in Engbert et al. (2011).** Data were recorded from fixational eye movements during a fixation of 2 s. Slow movements (blue) are highly erratic, whereas microsaccades (red) are ballistic, small-amplitude epochs with a more linear trajectory (compared with the slow background motions). The sample trajectory was recorded with a sampling frequency of 500 Hz (for details see ref. 29 in Engbert et al., 2011). Reprinted from Engbert et al. (2011), copyright PNAS 2011.

Brownian motion, this relation is linear in time. This characterization has been adopted in some recent models of how vision may cope with (Pitkow et al., 2007; Burak et al., 2010), and indeed exploit (Kuang et al., 2012), jitter of the retinal image due to FEM.

At the motor end, an important neural correlate of oculomotor postural control is *persistent neural activity* in brainstem regions including the prepositus hypoglossi ("PH") (Delgado-Garcia et al., 1989) and medial vestibular nucleus ("MVN") (Serafin et al., 1991; du Lac and Lisberger, 1995). Persistent here refers to sustained activity on timescales much longer than individual neural spiking timescales, in response to a relatively brief stimulation. Seung (1996) described a model of how persistent neural activity could be maintained through positive feedback, and showed how FEM drift-tremor could reflect a random walk along the line attractor created by the positive feedback dynamics in the motor memory of eye position. Seung suggested that various sources of noise, such as the random fluctuations in the tonic input from vestibular afferents, could be causing the random walk behavior.

Persistent neural activity is associated with short term memory more generally (Major and Tank, 2004), and various potential mechanisms for maintaining persistent activity have been investigated (see for review Brody et al., 2003). Stability issues arising from the positive feedback model were addressed in Koulakov et al. (2002); Goldman et al. (2003). More recently, empirical evidence contrary to the predictions of line attractor models (Aksay et al., 2007; Miri et al., 2011a) has motivated the proposal of modifications of recurrant network models, and the development of new models based on functionally feedforward networks and balanced regimes of excitation and inhibition (Goldman, 2009; Lim and Goldman, 2013). We address this topic in more detail in the following section.

#### *3.1.2. Self-avoiding random walk models*

Recent work has shown that FEM exhibit non-trivial temporal correlations whose description require fractional scaling exponents, rather than the unitary scaling exponent of pure Brownian motion. Engbert and Kliegl (2004); Mergenthaler and Engbert (2007) applied random walk analysis to the statistics of fixational eye movements at short (*<*40 ms) and long time (100–400 ms) scales. The tremor component was not resolved in these studies. At short timescales, the fixation point drifts faster than for normal diffusion/Brownian motion, with scaling exponents in the range 1.3–1.5. This is termed a "persistent" Codling et al. (2008) or "superdiffusive" Metzler and Klafter (2000) process. On longer timescales, however, the distance of the fixation point drifts slower than normal diffusion (termed subdiffusive/antipersistent). We adopt "sub/superdiffusive" here to avoid crossing terminology with that of persistent neural activity.

(Engbert et al., 2011) modeled this behavior in terms of a self-avoiding random walk ("SARW") in a potential. The potential accounts for the long term subdiffusivity, implementing a tendency for "wandering back" to the center of the potential well in the long term. The short term superdiffusivity is modeled by giving the random walk a memory, and adding a term such that the walk next visits the neighbor whose "previously visited" activation is lowest, with ties resolved randomly. As a result, the walk tends to be structured by the initial direction which is chosen (randomly), as the path backwards is in general more visited. In this model, microsaccades are triggered by visiting a site with a "visited" activation above a threshold. Roberts et al. (2013) describe a similar self-avoiding random walk model, though without the confining potential and the microsaccade threshold, and with an imprecise continuous memory and rather than a lattice representation. Importantly, this study showed that both the superdiffusive drift in the FEM signal, and the selfavoiding random walk approach, generalize to more ecologically valid, dynamic viewing conditions (watching a film).

#### *3.1.3. Recent findings may challenge some existing ideas about FEM*

Recently, Zhang and Li (2012) reported technical innovations based on binocular video tracking of ocular vein structure, which enable non-invasive, high resolution imaging of FEM in three rotational degrees of freedom. Importantly, the optical imaging approach avoids the interaction of mechanical measurement devices with ocular tremor. The authors reported a previously unsuspected level of structure at high resolution. See **Figure 3** for a FEM trajectory where tremor is not resolved. **Figure 4** depicts a FEM trajectory as measured by this new technique. These findings are quite new and have not received much attention to date, at least in terms of citations. They likely merit more attention, because if they are reproducible, they offer a challenge to existing conceptions of FEM in a number of ways.

Firstly, the microsaccadic trajectories recorded in these high resolution data are not straight and ballistic, as is widely supposed Martinez-Conde et al. (2004); Martinez-Conde (2006); Rolfs (2009); Martinez-Conde et al. (2013). The microsaccadic trajectories observed by Zhang and Li (2012) were often curved, and could exhibit fine scale changes in both speed and direction. Secondly, and crucially to our argument here, Zhang and Li (2012) reported that drift can take the form of a curled line at a fine spatial scale wherein tremor is resolved. This observation of a trajectory which is habitually self-crossing at a small spatiotemporal scale is not predicted by self-avoiding random walk models of drift generation, as the process should be self-avoiding at small spatiotemporal scales. These findings are a significant part of the motivation for the current hypothesis. Unfortunately, however, we have as yet been unable to obtain the associated time series data. Speculativeness notwithstanding, we believe that the proposal of testable hypotheses is a positive way to motivate the publicization of data and structure further empirical investigations in this area. Another high resolution non-contact FEM measurement method which can resolve tremor has recently been reported by Kenny et al. (2013a,b), so data of sufficient resolution may soon become available from this group.

#### *3.1.4. Tremor; a clue to the mechanisms of FEM, or "just noise"?*

Tremor has often been dismissed as "noise," but then so have other aspects of FEM over the years. Whether tremor reflects unrelated background noise or the workings of the neural mechanisms which maintain the dynamical posture of the eye remains very much an open question. Spauschus et al. (1999)found strong binocular coherence of tremor, and concluded that tremor reflects the patterning of low-level drives to oculomotor neurons, rather than motor noise. More recently, a sophisticated method was employed by Thiel et al. (2008), who reported positive evidence for binocular phase synchronization. They concluded that there might be only one center in the brain that produces the fixational movements in both eyes, or a close link between the two centers. The loss or reduction of tremor in certain cases of brain pathology (Michalik, 1987) and in coma (Shakhnovich and Thomas, 1977) also gives reason to suspect a more important, and delicate, source of tremor.

#### **4. MOTIVATIONS FOR THE SPIRAL WAVE HYPOTHESIS OF FEM DRIFT-TREMOR**

#### **4.1. STATISTICAL SIMILARITIES BETWEEN DRIFT-TREMOR TRAJECTORIES AND SPIRAL DYNAMICS**

Tremor contains a strong spectral peak around 80–90 Hz and a less prominent, variable lower frequency component up to around 25 Hz (Spauschus et al., 1999). A low amplitude (approx. 1 photoreceptor), slow (around 5 Hz), tremor-like ocular oscillation has been recorded at very high resolution in the turtle (Greschner et al., 2002), possibly corresponding to the slow component of tremor in primates, though cross species comparisons must be made with caution (Martinez-Conde and Macknik, 2008). Greschner et al. (2002)reported this oscillation as periodic, though the flat peak in the frequency spectrum around 5 Hz, and the high variability visible in the inset example trajectories, may be suggestive of quasi-periodicity (see their Figure 1A). Either way, the regularity of this low frequency component is interesting because it suggests a certain systematicity to the generative mechanisms. Closer examination of human tremor is required to establish whether an identifiable carrier wave exists at lower frequencies, which is then heavily masked by the fast aperiodic component of tremor. An underlying spiral wave neurodynamics is consistent with both a periodic and a quasi-periodic form for this carrier wave (Barkley et al., 1990; Broer et al., 1996), and predicts the accompanying high frequency aperiodic oscillation (see **Figures 1**, **2**).

See **Figure 5**, reprinted from Figure 7 in Huang et al. (2010), for visual representations of spiral drift recorded in visual cortex. Like FEM drift (Cornsweet, 1956; Matin et al., 1970; Findlay, 1971), spiral drift can exhibit Brownian statistical structure, due to both external forcing/environmental gradients (Sendiña-Nadal et al., 2000; Yuan et al., 2011) and intrinsic dynamics (Biktashev and Holden, 1998). Such gradients could (but need not) reflect the path memory and/or confining potentials in SARW models (Engbert et al., 2011), and perhaps even visual context (Mensh et al., 2004; Chan and Galiana, 2005). Like FEM drift, the velocity of spiral drift in neocortex is variable (Huang et al., 2010), and was found to be higher in induced sleep-like states (see **Figure 5**), consistent with recent observations that time-on-task increases the speed of FEM drift, whilst reducing the peak velocity of microsaccades (Di Stasi et al., 2013).

*Prediction* The singularity hypothesis predicts that, when the small spatial scale of tremor is resolved, drift-tremor trajectories will take a curling, self-crossing form reflecting the rotational orbit of a spiral wave. This might also be complex due to interactions between similar systems controlling different degrees of freedom. The fast component of tremor will manifest as an aperiodic modulation of this carrier wave. The presence of a slow (10–20 Hz) rotational component, giving the trajectory a habitually *self-crossing* form at small scales, would distinguish the spiral wave model from self-avoiding random walk models of FEM drift.

#### **4.2. SPIRAL WAVES AND THE NEURODYNAMICS OF OCULOMOTOR POSTURAL CONTROL**

#### *4.2.1. Persistent neural activity*

*Persistent neural activity* ("PNA") refers to localized "bumps" of fast firing cells which persist over timescales much longer then the timescales of the individual neurons comprising the bump. PNA is an important neural correlate of working memory (Major and Tank, 2004), and has been studied intensively in the context of brainstem *neural integrators* which encode eye position during oculomotor control (e.g., Aksay et al., 2001, 2007; de Dios Navarro-López et al., 2004; Miri et al., 2011a). Fluctuations in oculomotor PNA are thought to underly FEM drift and tremor,

test, *p <* 0*.*001, 25 *t* tests). The difference between *in vivo* (Cch/bic) and *in vivo* (sleep-like) is also significant. Reprinted from Figure 7 in Huang et al.

but the precise mechanisms responsible for their generation remain a matter of considerable debate. Current models regarding these mechanisms have been categorized on the basis of whether they posit intrinsic unicellular mechanisms (e.g., Shen, 1989; Loewenstein and Sompolinsky, 2003; Teramae and Fukai, 2005) or network mechanisms (Cannon and Robinson, 1985; Seung, 1996; Seung et al., 2000; Goldman et al., 2003; Goldman, 2009). Some role for network mechanisms is suggested by evidence of correlated activity between cells in the oculomotor integrator network (Aksay et al., 2003), and by the observed covariance of eye position and the frequency and magnitude of the synaptic barrage converging on integrator cells (Huang, 2009). These alternatives are not necessarily mutually exclusive, and multiple mechanisms may operate in the maintainance of PNA in the oculomotor system and elsewhere in the brain (Major and Tank, 2004).

The vestibulo-oculomotor system exhibits fractional dynamics (Anastasio, 1994), and complex time variation in PNA has motivated arguments that a model with multiple timescales of persistent firing may be required (Anastasio, 1998). Indeed, recent evidence for multiple timescales of persistence in oculomotor PNA (Miri et al., 2011a) suggests a higher dimensional attractor dynamics than proposed by earlier line attractor models (Seung, 1996; Seung et al., 2000; Goldman et al., 2003), leading to the development of new models with more complex dynamics (Miri et al., 2011a; Fisher et al., 2013). (Goldman, 2009) describes a functionally feed-forward architecture which reproduces some of the time variation in PNA, showing that positive feedback is not essential in principle, while Lim and Goldman (2013) presents a model based on homeostatic mechanisms which maintain a careful balance of excitation and inhibition. In human psychophysical studies, Khojasteh et al. (2012) found that cross subject averaging hides idiosyncratic nonlinear patterns. All this suggests that considerable complexity inhabits the dynamics of PNA in the oculomotor system (Durstewitz and Seamans, 2006).

The current hypothesis suggests an addition to the repertoire of hypothesized mechanisms for PNA, which falls into the category of network mechanisms, though is distinct from existing network models in a number of ways. Unlike existing network models, which concentrate on modeling neural behavior at the level of firing rates and drift, we focus on the finer spatiotemporal scale of FEM tremor and subthreshold fluctuations in the membrane potential of cells mediating oculomotor integration. Rather than specific circuit design, persistence is based on the transient self-organization of population activity into a reentrant, (quasi)periodic spatiotemporal pattern. Spiral waves require a predominance of excitatory, spatially distributed connections but precise connectivity structure is not required; spirals can easily emerge in randomly connected networks (Milton et al., 1993; Chu et al., 1994; Yuan et al., 2011). This is not to say that specific circuitry is not important or present in the hVPNI; just that it is not a requirement of the current model. The existence of spatially organized, excitatory lateral connectivity is suggested by various studies (Aksay et al., 2001, 2007; Miri et al., 2011a). Disinhibition is crucial to the formation of spirals (and other traveling waves) in cortical tissue (Huang et al., 2004, 2010). (Aksay et al., 2007) identified that mutually inhibitory collateral interactions were not necessary to local integrators within a certain range, suggesting

(2010), copyright Elsevier 2010.

that these mutually inhibitory interactions regulated local mechanisms rather than driving PNA directly. Such a situation could relate to mutual modulation of the excitability required for the emergence and propagation of traveling waves.

#### *4.2.2. Functional properties of spiral waves*

Certain features of spiral waves make them potentially interesting as neurodynamical mediators of PNA. Indeed, spiral waves are a form of neural activity which is persistent (e.g., Milton et al., 1993; Chu et al., 1994), though they are not usually associated with the term as used in the context of short term memory and neural integrators. The combination of A, B, C and D below suggests a mechanism capable of contributing to the maintenance of localized persistent neural activity.

*A. Spatiotemporally organized depolarization:* Sub-threshold traveling waves in the membrane potential field coordinate network activity in space and time, by defining spatiotemporal regimes of polarization-depolarization (Wu et al., 2008; Huang et al., 2010; Bahramisharif et al., 2013). Cells in the vicinity of the singularity can take arbitrarily differing phase, leading to an almost phaseless depolarizing synaptic barrage in that vicinity.

*B. Pseudo-locality:* Spiral waves exhibit a duality which gives them both a local, particle-like description (the singularity) and a global wave-like description (the propagating spiral arms) (Biktasheva and Biktashev, 2003). Though the wave is extensive, its behavior is almost entirely based on what happens in the neighborhood of the singularity.

*C. Quasi-persistence:* Spirals are reentrant waves, whose activity generates the conditions for their own persistence in time (Winfree, 1991), subject to certain conditions (e.g., Ito and Glass, 1991; Fenton et al., 2002; Zhang et al., 2003; Chun-Ni et al., 2010; Ma et al., 2010, 2012a). C combined with A and B, enables a spiral wave to *persistently* depolarize a *spatially localized* region in the neighborhood of the singularity.

*D. Seedability:* Spiral waves can be induced in an appropriate medium by various methods (Aranson et al., 1994; Williams and Holland, 1999; Leanhardt et al., 2002; Zhang et al., 2002; Xiao-Ping et al., 2011; Yuan et al., 2011; Ma et al., 2012b).

The most effective and tunable method is probably to directly impose an external forcing spiral, as in (Xiao-Ping et al., 2011), or a spiral seed plus a periodic forcing current near the singularity (Zhang et al., 2002). However, persistent spirals can also be induced in networks of integrate-and fire neurons by a brief, non-spiral periodic forcing (Milton et al., 1993; Chu et al., 1994; Huang et al., 2004; Kilpatrick and Bressloff, 2010b; Yuan et al., 2011) given some reasonable connectivity conditions (chiefly spatiality and some kind of inhomogeneity/noise/perturbation which breaks rotational symmetry).

D combined with A, B and C, provides a mechanism whereby an afferent may seed a spatial pattern in an efferent, and then leave that pattern to sustain itself with a certain amount of autonomy. Durstewitz and Deco (2008); Friston et al. (2012) suggest that brain activity is characterized by a high dimension chaotic background state, from which lower dimensional metastable states transiently emerge. **Figure 6**, from (Zhang et al., 2002) nicely visualizes the notion of how a spiral wave seeding might realize such a transient dimensionality reduction in the context of the

**FIGURE 6 | Examples of spiral seeding in a background of chaotic turbulence.** Seeding and growing spiral waves in a background of chaotic turbulence. The extent of the spiral wave seeded was dependent on the frequency of periodic forcing at the singularity. Too low, or too high, frequency was less effective. The ratio of the frequency of the forcing signal to that of the angular frequency of the spiral equalled **(A)** 0.6 **(B)** 0.8 **(C)** 1.0 **(D)** 1.2. Figure reprinted from Zhang et al. (2002). Copyright American Physical Society 2002.

observed (Aksay et al., 2001) difference between a background "off" state of the integrator, characterized by irregular firing at low rates (the turbulent background), and an "on" state characterized by driving input from the seeding of a spiral wave. de Dios Navarro-López et al. (2004) induced PNA in an oculomotor integrator circuit with brief, cholinergic periodic forcings. Oscillatory neurons observed in the guinea pig nucleus prepositus hypoglossi (Idoux et al., 2006), a region deeply associated with oculomotor integration (Delgado-Garcia et al., 1989; McCrea and Horn, 2006), might exemplify a neural substrate for periodic forcing inputs and the maintainance of traveling wave activity in the population. The persistent spiking of neurons in the vicinity of the induced singularity will be facilitated due to constant depolarization of the cellular membrane by high frequency microstimulation without a strong phasic component, as cells near the singularity may take arbitrarily different phase.

The bump of persistent firing activity in PNA has naturally been associated with "bumps" in neural field models (e.g., Tegnér et al., 2002; Owen et al., 2007; Kilpatrick and Ermentrout, 2013). However, it has also been suggested that hVPNI neurons may operate in a fluctuation dominated regime, in part because the firing threshold of the cell *increases with the membrane potential*, and in part because firing always occurs at the apex of membrane potential fluctuations (Huang, 2009). In a fluctuation dominated regime, firing rate is dependent less on mean membrane potential than on fast fluctuations in the level of the depolarizing synaptic barrage. On this view, at the level of subthreshold field dynamics fast microfluctuation (from a spiral wave) could be more effective than a constant raise in stimulation (from a bump) in inducing persistent firing. Thus spiral waves possess a unique repertoire of functional properties which render them interesting in the context of PNA. However, the presence or absence of a spiral wave only yields a binary distinction between an "on" state and an "off" state. Still missing is a mechanism for a graded temporal memory capable of remembering multiple, arbitrary step changes.

### *4.2.3. Possible mechanisms for continuous temporal integration*

Persistent neural activity is associated with *neural integration* (Major and Tank, 2004), and has been intensively studied in the anatomical context of oculomotor postural control, in particular the *horizontal velocity to position integrator* ("hVPNI") (Aksay et al., 2000). Here, a cell integrates (in the mathematical sense) its inputs over time, providing the ability to hold, and externally nudge, set points. An external nudge is reflected in a persistent shift in average membrane potential, co-occuring with step changes in firing rate and eye position (Aksay et al., 2001). How might this graded integration functionality be implemented by a spiral wave depolarization regime?

One possibility here is that spiral waves of different spatial extent generate different levels of depolarizing input. **Figure 6**, reprinted from Zhang et al. (2002) depicts how different intensities of forcing current can generate different sizes of spiral wave, from a pre-existing background of low level, chaotic turbulence. Increasing the size of the spiral by adding energy could perhaps encode up steps, but it is less obvious how a down step would be implemented. Another possibility is inducing multiple spirals.

Functionally, however, modulation of the frequency of the spiral's rotational orbit could provide the most appropriate variable for graded temporal integration. The excitability of the medium has a strong determining effect on the frequency taken by spirals and other traveling waves (Winfree, 1991). Modulation of the strength of lateral connections might therefore provide a mechanism to induce persistent changes in spiral frequency. Calcium mediated presynaptic facilitation (Mongillo et al., 2008) could provide a mechanism for strengthening lateral connectivity, and other modes of disinhibition could also be relevant (e.g., de Dios Navarro-López et al., 2004). Kilpatrick and Bressloff (2010a,b) describe spirals in neural field models, in which spike frequency adaptation modulates the frequency of network oscillations. Whatever the mechanism of frequency modulation, a faster spiral would generate more action potentials per unit time in the synaptic barrage converging on the cell from weak but numerous lateral connections, maintaining membrane depolarization, and the magnitude of these would be amplified by lateral synaptic facilitation. Both the magnitude and the arrival frequency of depolarizing excitatory postsynaptic potentials converging on active eye position coding cells varies systematically with eye position Aksay et al. (2001); Huang (2009).

*Secondary Prediction* A spiral frequency based temporal integrator is consistent with the close covariance of the arrival frequency of action potentials with eye position (Huang, 2009), and would predict in addition that a spectral peak in the slower range (around 10–20 Hz) of membrane potential oscillations during PNA will vary systematically with eye position in *individual trial* data (averaging might hide this effect). Note that the frequency modulation approach to graded integration is a secondary hypothesis.

#### *4.2.4. Subthreshold dynamics of the membrane potential during PNA in an oculomotor integrator*

Aksay et al. (2001) carried out *in vivo* intracellular recording and perturbation of persistent activity in an oculomotor neural integrator. They tracked the evolution of the cellular membrane potential during step changes in persistent activity associated with position control during fixation events. Their Figure 2 is reprinted here as **Figure 7**. See also their Figure 1 for longer recording period. Note the step like change in membrane potential that accompanies the onset of persistent firing. Further step changes are marked by brief (50–100 ms) overshoot/undershoot depending on direction of change, followed by a persistent change in the mean membrane potential and firing rate. The membrane depolarization was found to be sufficient to explain the associated PNA in a control experiment, suggesting an important role for network mechanisms.

Up (down) steps in membrane potential are correlated with increases (decreases) in both arrival frequency and magnitude of *excitatory postsynaptic potentials* ("EPSPs") (Aksay et al., 2001; Huang, 2009). The membrane potential (*Vis*)and the firing rate (*F*intra) shows signs of an oscillation at around 15 Hz, perhaps corresponding to the rotational orbit of a spiral wave and the slow component of tremor. If the barrage of depolarizing EPSPs are indeed originating from the slow rotational orbit and fast jitter of a spiral wave, then under close examination one would expect to see the leading edge of the spiral waveform reflected in the EPSPs. Huang (2009) examined membrane potential fluctuations during PNA in great detail. Whole-cell patch recordings revealed the existence of many small (0.2–3 mV) excitatory postsynaptic potentials lasting 5–10 ms, and manifesting a "peculiar" sharp-attack, slow-decay form obscured in accompanying sharp electrode recordings. See their section 5 and Figure 5.3, reproduced here as **Figure 8**. This waveform is typical of that generated by the passing of the leading edge of a spiral wave, where the phase gradient is very high (providing the sharp attack). See for example the depictions from Qu et al. (1999) of action potentials caused by cardiac spiral waves http://ajpheart.physiology. org/content/ajpheart/276/1/H269/F8.large.jpg. Note though that other possibilities exist. Based on similar waveforms observed in Mauthner cells (Golding and Spruston, 1998; Korn and Faber, 2005), Huang (2009) suggests that mixed NMDA/AMPA conductances could underly the shape of these potentials.

*Prediction* At the network level, the singularity hypothesis predicts that spiral waves should be directly identifiable in the VSDI signal at the site of PNA (at least in the anatomical context of brainstem oculomotor integration), and that the success (failure) to induce a spiral wave will distinguish success (failure) to induce PNA. While it is possible in principle that the membrane potential dynamics just reviewed could reflect a spiral wave which exists elsewhere, neurophysiological evidence suggests that the mechanisms sustaining PNA in the hVPNI are local (Aksay et al., 2007).

A standard test for whether an observation corresponds to a true spiral wave is the existence of a phase singularity, with a local amplitude reduction of field oscillations in the vicinity of the singularity (Winfree, 1991, 2001; Huang et al., 2010), caused by the cancellation of signals from closely located nodes with opposing phase near the singularity. See Figure 1A in Huang et al. (2010), reprinted here as **Figure 9** for neurophysiological recordings of this phenomena. Small fluctations in the balance of this cancellation result in an increase in high frequency fluctuations visible in the VSDI signal in **Figure 9** (and also visible as jitter of the singularity in our **Figure 1**), which could drive a cell effectively in a fluctuation dominated regime (Huang, 2009).

We are not aware of VSDI studies of brainstem neural integrators. Recent methodological advances may combine to provide opportunities for imaging the spatiotemporal dynamics of subthreshold activity in the deep brainstem. Fiber optics offer a means to image non-superficial regions (Flusberg et al., 2005). Combining VSDI and laser scanning microstimulation offers a fast method for anatomical and functional mapping (Xu et al., 2010). Zebra fish larvae have recently been shown to provide an *in vivo* preparation with high optical transparency (Miri et al., 2011a; Fisher et al., 2013). Miri et al. (2011b) used two-photon laser scanning microscopy (Stosiek et al., 2003) to simultaneously image many cells in neural integrator circuits in the larval zebra fish, and introduced a semi-automated approach for identifying behavior measure (in this case eye movement) related cells in the ensuing space-time series.

Combining this methodology with VSDI, which would provide access to subthreshold spatiotemporal dynamics associated with PNA, could test directly whether spiral waves exist and if so, whether they are spatially associated with active eye position integrators and whether their rotational frequency (and/or spatial extent) covaries systematically with eye position. Regardless of whether these predictions are confirmed or denied, VSDI data would likely to be of great utility in the general research effort on the hVPNI.

#### **4.3. SACCADES AND MICROSACCADES**

The current contribution focuses on the dynamic maintainance of oculomotor posture between microsaccades. Nonetheless microsaccades, and indeed saccades in general, are naturally relevant to the discussion as a whole. In this section, we briefly address how a spiral wave model of drift-tremor might fit into its saccadic context.

#### *4.3.1. Scale free saccadic behaviors*

Recent evidence points to a remarkable continuity in the statistics of saccadic oculomotor control across scales including microsaccades (Otero-Millan et al., 2013). How might a spiral wave model of fixational postural control fit into its containing context of fast (micro and macro) saccadic gaze shifts? Is there a continuously

during intracellular recording. Solid lines at lower left indicate the average value of *Vis* during separate fixations. **(B)** Changes during off-direction steps

*(Continued)*

#### **FIGURE 7 | Continued**

of the ipsilateral eye. This segment is taken from the end of a nasaltemporalnasal cycle that started with the transitions shown in **(A)**. **(C)** Rate and potential changes for a different neuron during an on-direction step. The firing rate (*F*extra) of a second extracellularly recorded position neuron (*V*extra) served as a surrogate for eye position. In this recording, the fast afterhyperpolarization following action potentials was abolished by substituting cesium for potassium in the electrode solution. From Aksay et al. (2001), copyright Nature Publishing 2001.

**FIGURE 8 | Shapes of action potentials recorded with whole-cell patch measurements.** Note the "peculiar" triangular form, typical of the leading edge of a spiral waveform. From Huang (2009), copyright The Author.

scaling control principle which could give rise to a visual scan path with, from the bottom up;


The natural variability of human scanning patterns has been modeled at the macro-saccadic level by the imposition of a stochastic component comprising a Lévy walk (Klafter et al., 1987) upon scanpaths in a deterministic salience landscape (Brockmann and Geisel, 1999; Boccignone and Ferraro, 2012). Another possibility is that the stochastic component reflects an intrinsic probabilistic feature of the salience function, (e.g., Harel et al., 2006), rather than an imposed randomization.

#### *4.3.2. Lévy walks in rotational and turbulent flow*

Theoretical work has revealed deep links between spiral waves, turbulent flow, fractional Brownian motion, anomalous diffusion and Lévy type trajectories (Shlesinger et al., 1987; Viecelli, 1990; Metzler and Klafter, 2000). Solomon et al. (1993) observed Lévy walks of tracer particles in a physical system of effectively two dimensional rotating flow, and Solomon et al. (1994) examined in more detail behavior in periodic, chaotic and turbulent conditions. See our **Figure 10** for a reprint of Figure 6 in Solomon et al. (1994), depicting example trajectories. Long term trajectories exhibited a pattern of long, relatively direct flights in predominantly translational flow, interspersed with episodes where the particle is caught up in a spiraling curve due to capture by a vortex ("sticking"). Analysis of sticking times and flight statistics indicated a Lévy walk trajectory evolving in continuous time. Biomechanical constraints suggest that a *truncated* Lévy walk Mantegna and Stanley (1994), where maximum step lengths are finite, may be more appropriate to the biological case.

Similar dynamics in a more complex landscape consisting of multiple clusters of vortices could conceivably result in a trajectory resembling that of multiscale visual exploration. "Fixation" periods consist of clusters of mini-fixations, each of which consists of an episode of vortex sticking characterized by rotational flow. Escaping a cluster results in a relatively long step to the next cluster (i.e., a macrosaccade), followed by a sequential sampling of the new cluster. The traveling wave accompanying saccadic execution observed in the superior colliculus by Munoz et al. (1991) might be a manifestation of relatively long flights between vortex sticking visible in **Figure 10**. The curling trajectory of drift-tremor during microfixation and the curving, interrupted microsaccade trajectories reported by Zhang and Li (2012) are reminiscent of the turbulent transport scenario just outlined, though closer analysis of these data is required. Note how the microsaccadic trajectories in **Figure 4** do not always start their trajectory in the direction of their final destination. They are often curved and can have small scale variations in velocity. Looking closely at the microsaccades depicted, one may observe vertical motions exhibiting an oscillation which is damped in one horizontal dimension and amplified in the other, suggestive of transport in a potential field. Where they do travel in straight lines, this is usually on the horizontal axis, suggestive of

a potential field. Overall the pattern is not ballistic, but is consistent with the trajectory first escaping local vortex sticking in an unpredictable direction, followed by a trajectory dominated by translational flow with potential fields. **Figure 11** depicts an example of a trajectory in a model of anomalous transport in magnetic field turbulence, taken from Chiaravalloti et al. (2006). This provides an example, though from a different domain, of how these dynamics could generate trajectories similar to the FEM trajectories depicted in **Figure 4**. Adding a potential field might further approximate the FEM trajectories in **Figure 4**.

From Solomon et al. (1994), copyright American Physical Society 1994.

#### **4.4. TURBULENT TRANSPORT AND CHAOTIC ITINERANCY**

Thus there is some potential for a continuous, deterministic dynamical principle capable of generating the scale free, stochastic profile of visual scanning trajectories. This speculative proposed framework for multiscale visual exploration would imply a widespread role for traveling waves and rotational flow in brainbody hermeneutics, which may stretch the readers credulity at this stage, but there is some existing context. Breakspear (2001); Tyukin et al. (2009); Friston et al. (2012) examine traveling wave processing and self-organized instability in perception, whilst Heitmann (2013) explores traveling wave functionality in the motor context. In addition to noting a potential contribution to persistent neural activity in the sensory context, Huang et al.

IOP publishing.

(2010) suggests that spiral waves, as a locally generated event, may also help a local cortical circuit to quickly disengage from globally synchronized rhythms. If traveling waves are playing functional roles in brain activity, one role of rotational flow may be to "hold the posture" of the central nervous system, while translational flow interconnects metastable postural transients. If so, this should be reflected in fixational drift-tremor and saccades, because the eye is part of the CNS. Breakspear et al. (2010) suggest that traveling wave solutions may offer optimal solutions to minimization of the free-energy in far from equilibrium initial conditions. Free energy minimization may be a rather general heuristic in nervous function (Friston, 2010). If the predictions of the singularity hypothesis of FEM drift-tremor turn out to be accurate, then the case for transient population dynamics as optimizers of behavior would gain a considerable boost.

It is interesting to speculate that attention may be related to nervous mechanisms of suppressing spiral waves. On this view, the sequential visiting of spirals in a cluster would destabilize the spirals and cause their breakup, resulting in a collateral effect resembling inhibition of return. This kind of self-destabilizing, itinerant trajectory would link action and perception into a common framework probably best described in terms of existing work on (embodied) *chaotic itinerancy* (Tsuda, 1991, 2009; Kaneko, 1992; Kaneko and Tsuda, 2003; Ikegami, 2007). Transient dynamics traversing a landscape of attractor ruins with riddled basins (Milnor, 1985) (i.e., quasi-attractors whose basin of attraction is riddled with repellent trajectories belonging to the basin of another attractor) can perform perceptual (Breakspear, 2001; Tyukin et al., 2009) and memory (Rabinovich et al., 2001) functions. This raises the possibility that the transient dynamics of embodied eye movements could play a rather sophisticated perceptual role analagous in computational description to that of neural sensory mechanisms (Tyukin et al., 2009), but at the embodied level where the perturbation structure of the world constitutes the data set (Ikegami, 2007). Wilkinson et al. (2011) give a simple computational example of how exploratory gaze patterns structured by spiral waves can enact perception of a global property of a social scene (co-orientation), as has been observed in infants (Augusti et al., 2010; Handl et al., 2013).

#### **5. CONCLUSION**

Despite extensive study of oculomotor postural control, the generative mechanisms of fixational drift and tremor remain uncertain. We have proposed the hypothesis that these components reflect the drift and meander of spiral wave neurodynamics. Whilst speculative, the singularity hypothesis offers a parsimonious and predictive account of FEM. Though our motivations are chiefly functional, the available psychophysical and neurophysiological evidence is largely consistent with, and occasionally suggestive of, a contribution of rotational flow to the maintainence of persistent neural activity in the oculomotor system. We have laid out an argument motivating our hypothesis in terms of the existing literature, and made testable predictions which could falsify it. Our hope is that these predictions will encourage other groups working in the fields of FEM and oculomotor integration to consider looking for indicators of rotational flow when analysing data, and perhaps even motivate neuroimaging studies to examine the subthreshold spatiotemporal dynamics associated with PNA. Should empirical studies and/or analysis of existing data confirm the basic predictions of the conceptual model, future work should undertake detailed computational modeling. Further testing and development may offer a deterministic account of the stochasticity and self-similarity manifest in eye movement patterns across scales, based on the complex dynamics of anomalous transport in rotating neural flow.

#### **ACKNOWLEDGMENTS**

This research was supported by the EU project RobotDoc under 25065 from the 7th Framework Programme, Marie Curie Action ITN. We thank the reviewers for motivating significant improvements in the paper.

#### **REFERENCES**


architecture of a short-term memory microcircuit. *Neuron* 79, 987–1000. doi: 10.1016/j.neuron.2013.06.041


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 06 December 2013; accepted: 09 February 2014; published online: 26 February 2014.*

*Citation: Wilkinson NM and Metta G (2014) Capture of fixation by rotational flow; a deterministic hypothesis regarding scaling and stochasticity in fixational eye movements. Front. Syst. Neurosci. 8:29. doi: 10.3389/fnsys.2014.00029*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2014 Wilkinson and Metta. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# State-Dependent Propagation of Neuronal Sub-Population in Spontaneous Synchronized Bursts

Yuichiro Yada1, 2, 3, Ryohei Kanzaki 1, 2 and Hirokazu Takahashi 1, 2 \*

<sup>1</sup> Research Center for Advanced Science and Technology, The University of Tokyo, Tokyo, Japan, <sup>2</sup> Department of Mechano-Informatics, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan, <sup>3</sup> Japan Society for the Promotion of Science, Tokyo, Japan

Repeating stable spatiotemporal patterns emerge in synchronized spontaneous activity in neuronal networks. The repertoire of such patterns can serve as memory, or a reservoir of information, in a neuronal network; moreover, the variety of patterns may represent the network memory capacity. However, a neuronal substrate for producing a repertoire of patterns in synchronization remains elusive. We herein hypothesize that state-dependent propagation of a neuronal sub-population is the key mechanism. By combining high-resolution measurement with a 4096-channel complementary metal-oxide semiconductor (CMOS) microelectrode array (MEA) and dimensionality reduction with non-negative matrix factorization (NMF), we investigated synchronized bursts of dissociated rat cortical neurons at approximately 3 weeks in vitro. We found that bursts had a repertoire of repeating spatiotemporal patterns, and different patterns shared a partially similar sequence of sub-population, supporting the idea of sequential structure of neuronal sub-populations during synchronized activity. We additionally found that similar spatiotemporal patterns tended to appear successively and periodically, suggesting a state-dependent fluctuation of propagation, which has been overlooked in existing literature. Thus, such a state-dependent property within the sequential sub-population structure is a plausible neural substrate for performing a repertoire of stable patterns during synchronized activity.

#### Edited by:

Ruben Moreno-Bote, Universidad Pompeu Fabra, Spain

#### Reviewed by:

Artur Luczak, University of Lethbridge, Canada Victor De Lafuente, Universidad Nacional Autónoma de México, Mexico

#### \*Correspondence:

Hirokazu Takahashi takahashi@i.u-tokyo.ac.jp

Received: 27 January 2016 Accepted: 14 March 2016 Published: 31 March 2016

#### Citation:

Yada Y, Kanzaki R and Takahashi H (2016) State-Dependent Propagation of Neuronal Sub-Population in Spontaneous Synchronized Bursts. Front. Syst. Neurosci. 10:28. doi: 10.3389/fnsys.2016.00028 Keywords: spontaneous synchronized burst, state-dependent activity, microelectrode array, dissociated culture, metastable dynamics

# INTRODUCTION

Repeating stable spatiotemporal patterns emerge in synchronized spontaneous activity in vivo (Lee and Wilson, 2002; Ji and Wilson, 2007; Luczak et al., 2007; Villette et al., 2015), in vitro (Beggs and Plenz, 2003, 2004; Ikegaya et al., 2004), and in dissociated cultures (Segev et al., 2004; van Pelt et al., 2004; Eytan and Marom, 2006; Madhavan et al., 2007; Rolston et al., 2007; Schroeter et al., 2015). The repertoire of such patterns can serve as memory, or a reservoir (Maass et al., 2002; Sussillo and Abbott, 2009) of information, in a neuronal network; moreover, the variety of patterns may represent the memory capacity in the network (Shew et al., 2011). Furthermore, spatiotemporal patterns in spontaneous activity are often similar to those of evoked activity against external events

**Abbreviations:** BFM, Burst feature matrix; SPP, Sub-population pattern; SPAW, Sub-population activation weight.

(Arieli et al., 1996; Tsodyks et al., 1999; Kenet et al., 2003; Luczak et al., 2009), suggesting that the variety of spontaneous patterns constrain the processing capacity of external inputs (Luczak et al., 2009; Villette et al., 2015). Our present interest is therefore the neural mechanism required to build a repertoire of stable spatiotemporal patterns in synchronized spontaneous activities.

Both theoretical and experimental studies have demonstrated that stable patterns emerge in a sequential structure of a neuronal network, where each synaptic connection is unreliable, yet synchronized activities of a particular sub-population reliably elicit another sub-population activity (Abeles, 1991; Aertsen et al., 1996; Diesmann et al., 1999; Ikegaya et al., 2004). Nevertheless, the way in which the repertoire of patterns is built in such a sequential structure remains elusive.

To address this problem, state-dependency of neuronal activity is a plausible neural underpinning (Buonomano and Maass, 2009). Recently, cortical spontaneous activities were characterized as having multiple "metastable states" itinerating in an activity dependent manner (Mazzucato et al., 2015). A particular state should continue during a quiescent period (Dranias et al., 2013, 2015; Ju et al., 2015) because cellular and synaptic properties governing states are likely to last without explicit spiking (Buonomano and Maass, 2009). Based on these studies, we hypothesize that (i) stable spatiotemporal patterns in synchronized spontaneous activity are generated by sequential activation of sub-populations, and that (ii) these patterns are generated in a state-dependent manner, whereby multiple metastable states can be defined as a finite continuous period.

In the present study, we test our hypotheses in dissociated neuronal cultures. To date, spontaneous activities in neuronal cultures have been well characterized with a microelectrode array (MEA) (Beggs and Plenz, 2004; Eytan and Marom, 2006; Madhavan et al., 2007). However, the spatial resolution of conventional MEA is insufficient to capture the whole activity in the neuronal network, potentially causing misestimation of population properties (Gerhard et al., 2011; Ribeiro et al., 2014). To overcome this technical pitfall, we use cuttingedge complementary metal-oxide semiconductor (CMOS) microelectrode arrays (MEAs) (Berdondini et al., 2009; Frey et al., 2010; Obien et al., 2014; Müller et al., 2015), which offer excellent spatiotemporal resolution for investigating neuronal networks in vitro (Gandolfo et al., 2010; Bakkum et al., 2013a; Panas et al., 2015). The CMOS MEA used in this study can simultaneously measure neural activities from 4096 sites within 2.67 × 2.67 mm<sup>2</sup> at a sampling rate of 7 kHz. The high-dimensional spatiotemporal activity patterns are then characterized by non-negative matrix factorization (NMF) (Lee and Seung, 1999; Leonard et al., 2015; Wei et al., 2015) in order to visualize whether and how sub-populations are sequentially activated in a state-dependent manner.

We demonstrate that cultured neurons obviously perform a repertoire of multiple spatiotemporal patterns in spontaneous synchronized activity, while different patterns share a partially similar sequence of sub-populations. This supports the concept that the network has invariant sequential structures of subpopulations. Additionally, similar spatiotemporal patterns appear consecutively, which suggests that pattern generation is state-dependent. Our experimental results provide compelling evidence that a repertoire of stable neural patterns is generated in a state-dependent manner.

# MATERIALS AND METHODS

# Cell Culture

All experimental protocols were approved by the ethical committee of the University of Tokyo and conducted in accordance with the "Guiding Principles for the Care and Use of Animals in the Field of Physiological Science" by the Japanese Physiological Society. The cell culture procedure was based on previous reports (Bakkum et al., 2013a) and was slightly modified. Cortices were dissected from E18 Wistar rats and dissociated by 0.25% trypsin-EDTA (Invitrogen) and trituration. For cell adhesion, the electrode area of the high-density CMOS MEA (3Brain, Biochip 4096S) was coated with a 20-ul drop of 0.05% polyethylenimine (Sigma) and then a 20-ul drop of 0.02 mg/ml laminin (Sigma). On the MEAs, 30,000–40,000 cells were seeded with cell plating media: 850 ul of NeuroBasal (Invitrogen) supplemented with 10% horse serum (HyClone), 2% B27 (Invitrogen), and 0.5 mM GlutaMAX (Invitrogen). After 24 h, the media were replaced with cell growth media: 850 ul of DMEM (Invitrogen) supplemented with 10% horse serum, 0.5 mM GlutaMAX, and 10 ug of sodium pyruvate. Cultures were maintained in an incubator at 37◦C and 5% CO<sup>2</sup> humidified atmosphere. Half the media were exchanged twice a week. For avoidance of evaporation and infection, the well on the chip was covered with a custom-made lid except for the period during the medium exchange (Potter and DeMarse, 2001).

# Recording with High-Density CMOS MEAs

Extracellular voltage was recorded using a commercialized highdensity CMOS MEA system (3Brain). Biochip 4096S (3Brain) contains 4096 electrodes; the scale of the electrode is 21 × 21 um, and the distance of neighboring electrodes is also 21 um. The electrodes are squarely located in the 2.67 × 2.67 mm area. Extracellular signals were simultaneously captured from the 4096 electrodes through the CMOS MEA interface, BioCAM4096 (3Brain), at a sampling rate of 7 kHz. They were recorded using BrainWave (3Brain) computer software. Ten minutes of spontaneous activities of five cultures at approximately 21 days in vitro (DIV) were recorded; their spontaneous activities at approximately 10 DIV were also recorded for comparison. Recording was performed outside the incubator. The recording space was shielded with a blackout curtain to avoid potential effects of ambient light (Imfeld et al., 2008) and maintained at 35–36◦C atmosphere.

# Spike Detection

Spikes were detected from recorded data by using a precise spike timing detection (PTSD) algorithm (Maccione et al., 2009) installed in BrainWave. The parameters for the PTSD algorithm were as follows: standard deviation factor, 10.0; peak life-time period, 2.0 ms; refractory period, 1.0 ms. The timing of each spike was assigned to the timing of its negative peak. The median of the peak amplitude of detected spikes was calculated at each electrode; spikes detected at electrodes with lower median peaks than the threshold were excluded from the following process. The threshold was the third quartile of the median peaks, which was qualified by manual inspection. Spike sorting was not performed in this experiment.

### Burst Detection

Synchronized bursts (Kamioka et al., 1996) were detected from recorded spontaneous activity by a slightly modified version of the existing adaptive algorithm (Bakkum et al., 2013b). If Nspike spikes occurred at all electrodes in total within less than T ms, the period was defined as a burst. The threshold time, T, is adaptively determined from the inter-spike interval (ISI) of each culture. The distribution of ISI typically forms bimodal shapes; an interval that takes the minima at the valley of the distribution is chosen as T. Here, we set Nspike as 200 because the number of recording channels was larger than the setup used in the original paper. Additionally, post-hoc processing was conducted for avoidance of burst fragmentation. The original method has excellent sensitivity in detecting small sizes of bursts; however, it separated a large burst into several small bursts in some cases. Thus, if an interval between two consecutive bursts was less than 100 ms, the two bursts were merged into a single burst.

# Evaluation of Burst Peak Amplitude Distributions

The maximum number of array-wide spikes in a 10-ms time bin during a burst was defined as the peak amplitude of the burst. Distribution of spontaneous burst peak amplitude was evaluated to check diversity of bursts. The distribution of bursts in denseplated cultures showed fixed-peak-amplitude bursts or "super bursts" at approximately 10 DIV, while they showed bimodal or long-tailed ones at approximately 20 DIV (Wagenaar et al., 2006b). Kurtosis of the distribution,

$$k = \frac{E[\left(\mathbf{x} - E[\mathbf{x}]\right)^4]}{E[\left(\mathbf{x} - E[\mathbf{x}]\right)^2]} - 3,$$

where x is the value obeying the distribution, and the chi-square goodness-of-fit test for normal distribution, were used to evaluate unimodality of the distributions.

# Sub-Population Pattern Extraction by Non-Negative Matrix Factorization (NMF)

The number of spikes occurred in 10-ms time bins were counted at all electrodes and a 4096 × 60,000 matrix was obtained. The matrix was defined as an observed matrix, Y. The width of the bin was chosen on the basis of the time step used in the previous report that observed multiple recursive spatiotemporal patterns in synchronized bursts (Madhavan et al., 2007). It was hypothesized that a part of the neurons in a network constitute a co-active sub-population in the temporal resolution; the sequential activation of such sub-neuronal populations generates repeatable spatiotemporal activity of neurons as synchronized bursts. In this model, the activities of sub-populations are captured as reproducible spatial patterns; we refer to them as sub-population patterns (SPPs).

It was assumed that each element of the observation matrix, yi,<sup>t</sup> (i= 1, 2,. . ., 4096;t= 1, 2,. . ., 60000), was sampled from the Poisson process with parameter si,<sup>t</sup> , which indicates an instantaneous firing rate,

$$p\left(\boldsymbol{\chi}\_{i,t}\right) = Poisson\left(\boldsymbol{\chi}\_{i,t}|\boldsymbol{s}\_{i,t}\right) \dots$$

It was additionally assumed that the firing rate of all 4096 electrodes at each bin was generated by a linear combination of D pieces of SPPs, where D is the dimension of network activity. Consequently, the instantaneous firing rate matrix, S (4096×60,000 matrix), was represented as the product of an SPP matrix, H (4096×D matrix), which contains an SPP at each column, and a sub-population activation weight (SPAW) matrix, W (D×60, 000 matrix), which contains the coefficients for linear combination,

$$\mathcal{S} = H \times W.$$

Notably, the SPPs could have overlapping electrodes in this model. The element of the observed matrix is the number of spikes and is thus non-negative. We thus define elements of the SPP matrix and SPAW matrix as also being non-negative. With this assumption, the SPP and SPAW matrices can be derived from the observed matrix using NMF (Lee and Seung, 1999). NMF with the generalized Kullback–Leibler divergence cost function actually assumes the Poisson generative model described above. Thus, we implemented the following optimization:

$$minimize \ D\_{KL} \ (Y|HW)\_{\cdot}$$

s.t.∀i,∀d,hi,<sup>d</sup> ≥ 0; ∀d,∀t,wd,<sup>t</sup> ≥ 0,

where generalized Kullback–Leibler divergence DKL between matrices is:

$$D\_{KL}(A|B) := \sum\_{m,n} \left( A\_{m,n} \log \frac{A\_{m,n}}{B\_{m,n}} - A\_{m,n} + B\_{m,n} \right).$$

The algorithm proposed by Lee and Seung (2001) was adopted to solve this problem. Open source MATLAB codes were used with slight modification<sup>1</sup> . The number of SPPs, D, was empirically determined to be 10 (d= 1, 2,. . ., 10). The initial values of the matrix elements were randomly set, and the iteration loop was implanted 500 times. The calculation was independently repeated ten times. The result with the minimum cost function among all trials was used for further analysis.

# Burst Pattern Classification

Bursts were classified into several classes to identify multiple recursive spatiotemporal patterns. In the present study, SPAWs during a bursting period were used as a burst feature matrix (BFM) to characterize the burst. First, however, time spans of bursts were adjusted from detected burst periods to compare spatiotemporal patterns. The initiation point of a burst was

<sup>1</sup>https://github.com/audiofilter/nmflib

defined as the first bin where ten spikes or more were observed in the whole network in a detected burst period.

All BFMs have the same length of time before the initiation points (pre-initiation length) and the same length of time after initiation pointsı (post-initiation length). Thus, the preinitiation length was determined to be 100 ms to avoid inclusion of previous bursts in a BFM. The post-initiation length was adjusted depending on the cultures to include the burst with the longest length from its initiation point. Then, BFMs were classified by correlation-based hierarchical clustering with some modifications from previous studies. The maximum peak of cross correlation was defined as the similarity between BFM A and BFM B to avoid the effects of the extraction of burst periods.

$$Corr(A, A) := \sum\_{l=1}^{L} \sum\_{d=1}^{D} A\_{d, l} A\_{d, l}$$

Similarity (A, B) = max k X L l = 1 X D d = 1 Ad,<sup>l</sup> Corr (A, A) 1/2 Bd,l−<sup>k</sup> Corr (B, B) 1/2 k = −L, L + 1, . . .., 0, . . ., L−1, L ,

where L is the length of the BFMs, and zero is inserted into the elements of BFMs if l − k< 0 or l − k > L. If BFM A and BFM B are identical, Similarity (A, B) = 1. Similarities between all pairs of the BFMs were calculated; the highest similar pair was grouped. An averaged BFM of the group was then used as a new BFM that represents the grouped bursts.

The above procedure was repeated until all BFMs were merged into one class group. Subsequently, the number of classes that maximize the contrast function (Beggs and Plenz, 2004) was determined as the optimal number of classes. However, if the largest class occupied more than 90% of all bursts, the next peak of the contrast function was selected as the number of the optimal class because misdetected bursts could form a small fraction in some cases.

## Sub-Population Sequence Analysis

Sequences of sub-population activity between different classes of bursts were compared. The comparison was conducted because the sequences are assumed to be partially invariant regardless of the classes of overall spatiotemporal patterns if there exists stable sequential propagation between sub-populations. First, five of the ten sub-populations with the largest peaks of SPAWs in the averaged burst were selected. The other five small SPAWs were excluded as burst-unrelated, or low contributing sub-populations. Then, spatiotemporal patterns of bursts were converted into a sequence of the timing when each subpopulation took the maximum SPAWs. One of the burst classes with the largest summation of SPAWs was defined as a template class. Moreover, the sub-population sequence calculated from the averaged template-class bursts was defined as a template sequence.

Here, partial similarity with respect to the template sequence was evaluated for (i) sub-population sequences in template classes, (ii) those in non-template classes, and (iii) randomized sequences. The partial similarity was measured according to two kinds of criteria, which were adopted with slight modification from a previous study of memory replay in the hippocampus (Lee and Wilson, 2002). The first criterion is the number of times of permutation of a sub-population pair order to perfectly match the template sequence. The proportion of subpopulation sequences that can match the template with the same or less than N permutation was used as the index of similarity, where N is a threshold value. The second criterion is the reproducibility of the sub-population pair order (duplet) or the sub-population trio order (triplet). A pair and a trio with the highest order consistency in all synchronized bursts were used as the duplet and triplet. The probability that the duplet or triplet was replayed within sequences was used as the index of similarity. Significance of partial similarity was statistically tested according to the similarity indices of actual sub-population sequences against those of randomized sequences. The number of randomized sequences was identical to the number of total (both the template class and non-template class) bursts. The significance was evaluated by the Mann–Whitney U-test.

# Sequence Randomization

In sub-population sequence analysis, as well as in burst class consecutiveness analysis, randomized sequences were used to test the statistical significance of actual data. Random real numbers between [0 1] were sampled from the uniform distribution and assigned to all elements of the sequence. The elements of the sequence were then sorted according to the assigned numbers, and this sorted sequence was compared with actual data.

# Evaluation of Burst Class Consecutiveness

Consecutiveness in sequences of burst classes was evaluated by probability of burst class transition. The probability that the same burst class was generated successively in actual data was compared with that of randomized burst-class sequences. One hundred randomized sequences were generated against each culture. The statistical significance of actual data was evaluated for each data, thereby testing the null hypothesis that the median of the probabilities in the randomized sequences is equal to the probability of the actual sequence by the one-sample Wilcoxon signed-rank test.

# Evaluation of Periodical Similarity in Burst Patterns

To evaluate the periodical appearance of spatiotemporal patterns in synchronized bursts, Fisher's g-statistic was used to test the significance of periodicity (Wichert et al., 2003). Fisher's g-statistic is defined as:

$$\mathcal{g} = \frac{\max\_{i} I\left(\alpha\_{i}\right)}{\sum\_{i=1}^{\left[N\_{\text{sample}}/2\right]} I\left(\alpha\_{i}\right)},$$

where I (ωi) is the periodogram of the signal to evaluate, Nsample is the sample size of the signal, and ω<sup>i</sup> is a discrete frequency of the signal, ω<sup>i</sup> = 2πi/Nsample i =0, 1, 2, . . .,- Nsample/2 . The significance level of Fisher's g-statistic was determined from the distribution:

$$P(\emptyset > \emptyset^\*) = \sum\_{m=1}^M (-1)^{m-1}\_{\left[\frac{N\_{\text{sample}}}{2}\right]} C\_m (1 - m\emptyset^\*)^{\left(\left[N\_{\text{sample}}/2\right] - 1\right)}$$

,

where M is the largest integer less than 1/g ∗ . Mean burst similarity according to (i) the difference of burst indices, and (ii) the difference of burst appearance time, from which linear components were subtracted, were used as I (ωi) in this study. For avoiding misestimation from small samples, the differences of burst indices with more than 20 pair samples and pairs of bursts appeared within 400 s were used.

# RESULTS

High-density CMOS MEAs captured spontaneous neuronal activity from five cortical networks. Ten-minute recording was performed in each culture in two developmental conditions: in a developed period–−20.8 ± 2.2 (mean ± SD) DIV—and in a juvenile period— 9.8 ± 0.8 DIV—for comparison. The number of available electrodes that detected action potentials with amplitude larger than the threshold was typically around 1000 channels. Synchronized bursts of cortical neurons were observed from the juvenile periods and stably persisted through development (Kamioka et al., 1996). The bursts were detected with the adaptive algorithm (Bakkum et al., 2013b); the average number of detected bursts was 139 ± 68 (mean ± SD) in a developed period, whereas it was 88 ± 54 in a juvenile period.

Consistent with a previous study (Wagenaar et al., 2006b), juvenile cultures exhibited fixed-peak-amplitude bursts, while developed cultures exhibited variable-peak-amplitude bursts. Representative 30-s spontaneous activities of juvenile and developed cultures are shown in **Figures 1A–D**. A juvenile network shows homogeneous spatiotemporal spiking activity (**Figures 1A,B**). In a developed one, however, bursts show heterogeneous activity (**Figures 1C,D**). The distributions of burst peak amplitude—the maximum number of array-wide spikes in a 10-ms time bin during a bursting period—show a single peak in a juvenile culture (**Figure 1E**), but a bimodal shape in a developed culture (**Figure 1F**). **Figure 1G** shows kurtosis of the burst-peakamplitude distributions. Kurtosis tends to drop and become apart from zero with development, except one culture (Culture #3), which was excluded from further analysis.

The chi-square goodness-of-fit test was performed for testing whether the burst-peak-amplitude distributions were represented with normal distribution. The developedperiod distributions disobeyed normal distribution (Culture #1, p = 3.215×10−<sup>8</sup> , mean ± SD: 484.8 ± 225.1; Culture #2, p = 4.656×10−10, mean ± SD: 250.7 ± 176.2; Culture #4, p= 0.005357, mean ± SD: 198.7 ± 88.1; Culture #5, p = 3.775×10−16, mean ± SD: 1437 ± 631), while juvenile-periods ones were characterized with normal distributions (Culture #1, p= 0.7668, mean ± SD: 46.36 ± 6.92; Culture #2, p= 0.5175, mean ± SD: 46.65 ± 7.34; Culture #4, p= 0.9448, mean ± SD: 137.1 ± 41.5; Culture #5, p= 0.09618, mean ± SD: 91.02 ± 18.23). These results demonstrate that bursts in a developed period differ from those in a juvenile period in terms of characteristic peak amplitude. We thus hypothesize that the developed cultures recruit variable neuronal sub-populations to produce different patterns.

Spontaneous spiking activity of the cultured neurons was decomposed into SPPs and SPAWs. First, the frequency of spikes occurring in 10-ms bins were calculated at each electrode. Then, the NMF algorithm (Lee and Seung, 2001) decomposed a 4096-dimensional spike frequency matrix into ten SPPs, which represented the spatial patterns of reproducibly co-activated electrodes and ten-dimensional SPAWs. The number of spikes detected at each electrode at each time bin was modeled as generated by the Poisson process with a latent parameter, which corresponded to a firing probability at the time. The Kullback– Leibler divergence NMF hypothesizes that the latent parameters of the Poisson process are a linear combination of the SPPs.

Non-monotonic repeating spatiotemporal patterns of synchronized bursts were observed from dimension-reduced activity obtained by NMF. The representative data in **Figure 1A** are decomposed into the temporal pattern of SPAW (**Figure 2A**) and SPPs (**Figure 2B**). The spatial distributions of co-activated electrodes in the SPPs did not always localized; some subpopulations had spatially localized activity patterns, while others had rather dispersed distributions, as illustrated in **Figure 2B**. Overall SPAW confirmed that synchronized burst patterns were reproducible both spatially and temporally. This finding is consistent with the previous study that demonstrated the stability of synchronized burst patterns in dissociated cultures (Eytan and Marom, 2006). However, the bursts in the first half (before 15 s) and those in the second half in **Figure 2A** appear to have different spatiotemporal patterns. For example, SPP #1 was recruited in the second half, but not in the first. Nevertheless, other SPPs seemed to be activated similarly in all synchronized bursts.

Spatiotemporal patterns of bursts were hierarchically clustered according to similarity of BFM and classified into several classes, as shown in the dendrogram in **Figure 2C**. The similarity matrix of bursts in **Figure 2D**, where indices of bursts are sorted according to the dendrogram in **Figure 2C**, suggests that temporal activation patterns of sub-populations in bursts are repeated. The number of classes was chosen to maximize the contrast function (**Figure 2E**) (Beggs and Plenz, 2004). The horizontal dotted line across the dendrogram in **Figure 2C** indicates the selected level of cut for the classes. Mean trajectories of SPAWs for each class are then illustrated in **Figures 2F,G**. Remarkably, temporal patterns of SPAWs in different classes seemed to be partially similar; spatiotemporal patterns in both classes were likely characterized as having

FIGURE 2 | Decomposition of high-dimensional neuronal activity and classification of synchronized burst patterns. (A) Low-dimensional activity of the neuronal network represented with sub-population activation weights (SPAWs). The same period of Figures 1C,D is illustrated. (B) Sub-population patterns (SPPs) of spontaneous activity of cultured cortical neurons obtained with NMF. The SPPs are shown as corresponding to recording electrodes configuration. (C) A dendrogram represents a process of hierarchical grouping of BFMs. The dotted horizontal line indicates a selected level of the grouping. (D) A similarity matrix of sub-population activation weights during synchronized bursts. (E) A contrast function for the dendrogram shown in (C). Asterisk indicates a maximum peak of the function. (F,G) Mean SPAWs within classified burst classes.

common sub-populations with partially identical temporal orders. This suggests stable unidirectional propagation of subpopulation across different classes. Such a sequential structure of neurons in spontaneous synchronized activity was suggested in previous reports (Eytan and Marom, 2006; Ham et al., 2008; Raichman and Ben-Jacob, 2008).

To evaluate partial similarity of spatiotemporal patterns among different classes, the sequences of sub-populations were compared. **Figure 3A** illustrates a schematic procedure of the analysis. Five out of ten sub-populations were selected according to the largest peaks of SPAWs in all averaged BFM to exclude effects from burst-unrelated or burst-less-related sub populations. Then, each burst was represented as a sequence of selected sub-populations. The burst class with the largest total SPAWs was defined as a template class; the sub-population sequence of averaged bursts in the template class was defined as a template sequence.

**Figure 3B** shows the probability that a sequence had partial similarity with the template sequence in the mean of pair permutation times. The threshold was set to two times here. Obviously, sequences in the template class had a higher probability of having partial similarity with the template sequence compared with randomly generated sequences (p = 0.02857 < 0.05). However, sequences in the non-template classes also showed higher probability than random sequences (p = 0.02857 < 0.05), although the median probability was slightly smaller than that of the template class. The findings in the permutation analysis were reconfirmed in the appearance probability of the same duplet/triplet order as shown in **Figures 3C,D** (p = 0.02857 < 0.05 in all comparisons). Thus,

FIGURE 3 | Similarity in partial sequence of sub-population activation between synchronized burst classes. (A) Illustration of the procedure to evaluate partial similarity between sub-population sequences of bursts. SPAWs during a burst were converted into a sequence of their peaks. Sub-population sequences of bursts were compared with the template sequence. Permutation times for matching refers to the way in which many pair permutations are required to match the template sequence. Duplet/triplet order matching indicates whether the order of two/three sub-populations matches the template sequence. (B) Probability that permutation times for matching is two or less. (C,D) Probability that the duplet (C) or the triplet (D) order matched the template. The most reproduced duplet or triplet was selected for analysis. The Mann–Whitney U-test. \*p < 0.05.

different spatiotemporal patterns in synchronized bursts were likely to have partially similar sequences of sub-populations.

Next, temporal consecutiveness of these spatiotemporal patterns was investigated. If spatiotemporal patterns in synchronized bursts are an active representation of hidden states of a network (Buonomano and Maass, 2009), temporally neighboring bursts should exhibit similar patterns. **Figure 4A** shows when each class of bursts appeared through recording, clearly demonstrating that the same class of bursts appeared consecutively. The probability of remaining in the same classes of bursts was significantly higher in the experimental data than in the simulation data with randomized orders of burst classes in all the test dishes (**Figures 4B,C**; Culture #1: p = 1.578×10−30; Culture #2: p = 9.466×10−30; Culture #4: 2.157×10−22; Culture #5: p = 1.578×10−30). In other words, a network of cortical neurons tends to maintain similar burst patterns for a certain period.

The consecutiveness in spatiotemporal patterns was also supported by the unsorted similarity matrix of bursts shown in **Figure 5A**. Similar clusters along the diagonal in the similarity matrix indicate that similar spatiotemporal patterns appeared in succession. **Figure 5B** illustrates the relationship between similarity of spatiotemporal patterns and the difference in burst appearance indices. Interestingly, not only did neighbor bursts share highly similar spatiotemporal patterns, but also the similar pairs of bursts seemed to appear periodically. From this observation, it was postulated that the combination of consecutiveness and periodicity can account for the burst similarity depending on the difference of burst appearance indices.

After subtracting the linear regression line, normalized autocorrelation of the similarity function (**Figure 5B**) was computed to visualize periodicity; then, clear periodicity was found in the autocorrelogram (**Figure 5C**). Fisher's g-statistic was calculated from the periodogram of the residual similarity function to evaluate the significance level of the largest component of the periodicity (**Figure 5D**; Wichert et al., 2003); **Figure 5D** indicates that similarity fluctuation in **Figure 5B** had 3 cycles, corresponding to 64.0 length of cycle. All cultures showed significant (p < 0.001) periodicity (Culture #1, p = 1.568×10−43; Culture #2, p = 1.222×10−<sup>5</sup> ; Culture #4, p = 2.270×10−<sup>3</sup> ; Culture #5, p = 4.829×10−<sup>7</sup> ), while the length of the cycle varied between cultures (Culture #1, 64.0; Culture #2, 27.7; Culture #4, 66.0; Culture #5, 38.6). The same analysis was tested against the relationship between similarity of spatiotemporal patterns and the difference in their appearance time (**Figures 5E–G**). A mean interval of 64 bursts, i.e., a length of the cycle in **Figure 5B**, corresponds to 195.1 (±26.0, SD) s, which appeared as a peak in **Figures 5E,F**. All cultures except one showed significant (p < 0.001) periodicity (Culture #1, p = 3.492×10−<sup>8</sup> ; Culture #2, p = 1.483×10−<sup>8</sup> ; Culture #4, p = 0.1338; Culture #5, p = 3.492×10−<sup>8</sup> ). Frequency of the similarity functions ranged between 0.01 and 0.02 Hz (Culture #1, 0.01 Hz; Culture #2, 0.175 Hz; Culture #5, 0.01 Hz).

**Figure 5B** indicates that spatiotemporal patterns may not be homogeneous during a period of the same class bursts. To address this possibility, similarity of bursts was quantified in the first two bursts in a given period (Start-Start), in the burst pairs at the start and end of a given period (Start-End), and in the successive burst pair at the transition of the period (Class

A-Class A) ( ¯ **Figure 6A**). Consequently, the Start pair exhibited slightly but significantly higher similarity than the Start-End pair (p = 0.003), while both the Start pair and the Start-End pair exhibited significantly higher similarity than the Class A-Class A pair ( ¯ p= 6.383×10−<sup>7</sup> , p = 3.858 × 10−<sup>6</sup> , respectively) (**Figure 6B**). Thus, spatiotemporal patterns of the same burst class gradually change with time, yet this within-class fluctuation is much smaller than the abrupt change at the transition of the burst classes.

Consequently, these results demonstrate that spatiotemporal patterns of bursts were not stochastically generated; rather, they were consecutively and periodically generated. Our results support hypothetical ideas that spatiotemporal patterns of bursts depend on hidden internal states of the network (Buonomano and Maass, 2009), and that the internal states spontaneously and recursively fluctuate between multiple "metastable" states (Durstewitz and Deco, 2007; Mazzucato et al., 2015).

# DISCUSSION

By combining high-resolution measurement with a 4096-channel CMOS MEA and dimensionality reduction with NMF, we investigated synchronized bursts of dissociated cortical neurons at approximately 3 weeks in vitro. We found that bursts had a repertoire of repeating spatiotemporal patterns, and different patterns shared a partially similar sequence of sub-population. These findings support the idea of propagation of neuronal subpopulations during synchronized activity (**Figure 7A**; Abeles, 1991; Ikegaya et al., 2004; Eytan and Marom, 2006). Furthermore, we found that similar spatiotemporal patterns tended to appear successively and periodically, suggesting state-dependent fluctuation of propagation (**Figure 7B**), which is overlooked in existing literature. Thus, such a state-dependent property within the sequential structure is a plausible neural substrate for performing a repertoire of stable patterns during synchronized activity.

# Methodological Significances

CMOS MEA is an emerging platform for capturing electrophysiological activity of neuronal networks. It is analogous to a movie with a spatial resolution at a cellular level and temporal resolution at a single action potential (Berdondini et al., 2009). Only a small population of neurons exhibit high activity, which plausibly play crucial roles in the network (Wohrer et al., 2013); overlooking these neurons may lead to misinterpretation of results in our experiments. Thus, to avoid such sampling bias, cellular-level spatial resolution is required in the measurement (Panas et al., 2015). Furthermore, because typical burst activities of our interests last only for a few 100 ms (Eytan and Marom, 2006), the temporal resolution

FIGURE 6 | Variation of spatiotemporal patterns during a period of the same class burst. (A) Similarity of the first two bursts in a given period (Start-Start pair), that of the burst pair at the start and end of a given period (Start-End pair) and that of the successive burst pair at the transition of the period (Class A-Class A¯ pair) were evaluated. (B) Similarity of Start-Start pair, Start-End pair and Class A-Class A pair. The red lines indicate the median of the similarities. The blue boxes are ¯ ranges from the 25th percentiles to the 75th percentiles. The whiskers are the ranges of the all similarity excluding outliers (the red crosses). Wilcoxon signed-rank test. \*\*p < 0.01. \*\*\*p < 0.001.

defined as a finite continuous period.

should be on the order of ms to appropriately characterize the pattern in bursts. CMOS MEA is the only measurement device available that meets both of these spatial and temporal requirements.

The dimensions of CMOS MEA data are inherently much higher than those of the functional SPPs (Baruchi and Ben-Jacob, 2004) of our interests. Appropriate dimensionality reduction is therefore helpful in identifying functional patterns. In the present study, we employed NMF to identify stably co-activated neuronal sub-populations. Originally, NMF was developed to extract characteristic parts, such as an eye, nose, and mouth, from facial pictures (Lee and Seung, 1999). The practical advantages of NMF are that there is no need for pre-processing (Peyrache et al., 2009; Lopes-dos-Santos et al., 2013), and that non-negative components extracted from the spiking activity patterns are intuitively interpretable, just like the facial parts in facial pictures (Lee and Seung, 1999).

# Repeating Spatiotemporal Patterns in Synchronized Spontaneous Activity

Consistent with the present results, previous studies showed that cortical cultures have a repertoire of repeating spatiotemporal patterns (Segev et al., 2004; Madhavan et al., 2007; Rolston et al., 2007). A bimodal burst-peak-amplitude distribution and bimodal spatiotemporal patterns of bursts were typically observed in our experiments. In terms of burst peak amplitude, developed cultures (around 21 DIV) exhibited a bimodal distribution, whereas young cultures showed a uni-modal distribution (Eytan and Marom, 2006; Wagenaar et al., 2006b; Madhavan et al., 2007). Similarity-based clustering also demonstrated that synchronized bursts could be classified into a few patterns (Segev et al., 2004); i.e., small and large bursts (Madhavan et al., 2007). Our results extend these findings in that both large and small bursts share similar activation sequences of sub-populations.

# State-Dependency and Spatiotemporal Patterns

Our results demonstrate the spontaneous itinerancy between different classes of spatiotemporal activity, suggesting that metastable states exist in innately isolated neuronal networks in vitro. Some intermediate states may also exist because similarity of burst patterns fluctuated continuously within a given state. Similar spontaneous transitions between metastable states were recently reported in the gustatory cortex in vivo (Mazzucato et al., 2015). Cortical activities were also characterized as two extreme states, i.e., desynchronized and synchronized states, with continuum of intermediate states (Harris and Thiele, 2011). These metastable states have a time scale of seconds or minutes. Thus, they are different from previously described metastable states in dissociated networks through development, which have a time scale of weeks or months and transit only unidirectionally (Pu et al., 2013).

A stable activity state of a neuronal network has been often mentioned as an attractor (Cossart et al., 2003; Wagenaar et al., 2006a). Classically, an attractor in the neural network was postulated as a memory of specific information (Hopfield, 1982). The classical attractor networks, however, are biologically implausible because the number of attractors is limited compared to the capacity of information (Maass et al., 2002) because the converging time into attractors (Maass et al., 2002; Rabinovich et al., 2008) and the effect of spontaneous activity (Kurikawa and Kaneko, 2015) are not consistent with experimental observation. Therefore, according to recent studies, it is more biologically plausible that transient metastable dynamics dominate neuronal activity (Durstewitz and Deco, 2007; Rabinovich et al., 2008).

Usually, repeating spatiotemporal activity is apprehended only as a visible sign of a metastable state (Haldeman and Beggs, 2005; Mazzucato et al., 2015) during which cellular and synaptic properties—e.g., the effect of short-term plasticity, slow inhibitory post-synaptic potentials, NMDA channel kinetics, etc.—are lasting, forming a so-called "hidden state" of neuronal networks (Buonomano and Maass, 2009). Such hidden states could account for consecutive appearances of similar bursts and can be considered an internal memory of a neuronal network. This internal memory is likely stronger than the short-term memory of external events, which is easily broken by bursts; i.e., internal memory (Dranias et al., 2013, 2015; Ju et al., 2015). Inhibitory interneurons may significantly contribute to selection of spatiotemporal patterns (Sasaki et al., 2014), depending on such hidden states.

In addition, our results demonstrate that spontaneous bursts may induce state transitions. Similarly, co-activation of some neurons trigger a state transition in the gustatory cortex (Mazzucato et al., 2015). These findings suggest that the hidden states in a neuronal network dominate spatiotemporal patterns of spontaneous activities, which in turn modulate the hidden states. Such an interaction between the hidden states and spontaneous bursts is a possible underlying mechanism of the metastable activity in neuronal networks.

# Sequential Propagation Structures during Synchrony

Our results imply that multiple spatiotemporal patterns are generated by a common stable propagation structure in the network (Raichman and Ben-Jacob, 2008). Signal transmission within such a stable structure might therefore depend on the hidden states. This conceptual framework of sequential structure is compatible with previous findings of a small group of "leader neurons," which activate at burst initiation, and hierarchical structures in the dissociated neuronal networks (Eytan and Marom, 2006; Ham et al., 2008). Our results are also consistent with in vivo experiments in that activity bursts across states have similar spatiotemporal patterns (Luczak et al., 2013). Taken altogether, such a modified model of synfire chain with statedependent fluctuation can account for both stability (Eytan and Marom, 2006; Panas et al., 2015) and multiple pattern generation (Segev et al., 2004; Madhavan et al., 2007; Rolston et al., 2007) in synchronized activities.

Existing models mostly overlooked state-dependent property to account for the variety of spatiotemporal patterns in the neuronal network. For example, a branching process is one of the convincing models (Beggs and Plenz, 2003). It demonstrates that cortical networks in vitro have metastable states, and that the critical branching process maximizes the number of metastable states (Haldeman and Beggs, 2005). Fixed propagation probabilities between neurons are postulated in these models, and spatiotemporal patterns are generated stochastically. However, this model is inconsistent with our finding that each spatiotemporal pattern does not randomly emerge; instead, it is repeated in a temporally consecutive manner. Further modeling with state-dependent properties would be one of the future directions.

# Spontaneous Spatiotemporal Patterns In vivo

The repeating spatiotemporal patterns in spontaneous activities have been observed not only in vitro (Beggs and Plenz, 2004; Ikegaya et al., 2004), but also in vivo (Luczak et al., 2007). Neuronal networks may transmit information as "neuronal packets" (Luczak et al., 2013, 2015), i.e., activity of neuronal sub-population, generating such stable patterns, which is consistent with our results. Memory replay in the hippocampus is extensively studied as a possible mechanism of memory consolidation during sleep (Lee and Wilson, 2002) and memory retrieval at awaking immobility (Takahashi, 2015; Villette et al., 2015). In the sensory cortex, Luczak et al. found similarity between spontaneous patterns and evoked ones. They hypothesized that a repertoire of evoked responses is a fraction of a spontaneous repertoire (Luczak et al., 2009). Spontaneous activity might be considered a prior distribution of sensory inputs (Berkes et al., 2011).

Nevertheless, in the network that experiences no external inputs, we demonstrated that repeating spatiotemporal patterns emerge in a state-dependent manner. Such network could have similar functional structures during spontaneous activity and stimulus-evoked activity (Pirino et al., 2015). Future extensive studies in both experimental and theoretical approaches are

# REFERENCES


required to elucidate the functions and mechanisms of these spontaneous properties.

# AUTHOR CONTRIBUTIONS

YY, RK, and HT designed research; YY performed research, composed programs and analyzed data; YY, RK, and HT wrote the paper.

# ACKNOWLEDGMENTS

This works was partially supported by Japan Society for the Promotion of Science (JSPS) through Grant-in-Aid for Scientific Research (26630089), Grant-in-Aid for JSPS Fellows (26-10399) and Asahi Glass Foundation.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Yada, Kanzaki and Takahashi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Decomposition of spontaneous brain activity into distinct fMRI co-activation patterns

# *Xiao Liu\*, Catie Chang and Jeff H. Duyn*

*Advanced MRI Section, Laboratory of Functional and Molecular Imaging, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA*

#### *Edited by:*

*Emili Balaguer-Ballester, Bournemouth University, UK*

#### *Reviewed by:*

*Huo Lu, Philadelphia College of Osteopathic Medicine, USA Daniele Marinazzo, University of Ghent, Belgium*

#### *\*Correspondence:*

*Xiao Liu, Advanced MRI Section, Laboratory of Functional and Molecular Imaging, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Dr., Bldg. 10, Rm. B1D-723A - MSC 1065, Bethesda, MD 20892-1065, USA e-mail: liux15@ninds.nih.gov*

Recent fMRI studies have shown that analysis of the human brain's spontaneous activity may provide a powerful approach to reveal its functional organization. Dedicated methods have been proposed to investigate co-variation of signals from different brain regions, with the goal of revealing neuronal networks (NNs) that may serve specialized functions. However, these analysis methods generally do not take into account a potential nonstationary (variable) interaction between brain regions, and as a result have limited effectiveness. To address this, we propose a novel analysis method that uses clustering analysis to sort and selectively average fMRI activity time frames to produce a set of coactivation patterns. Compared to the established networks extracted with conventional analysis methods, these co-activation patterns demonstrate novel network features with apparent relevance to the brain's functional organization.

**Keywords: resting-state network, non-stationary connectivity, network dynamics, clustering, dynamic connectivity**

## **INTRODUCTION**

A growing body of neuroimaging research is reporting on the phenomenon of spontaneous neural activity occurring during rest, in the absence of overt behavior (Biswal et al., 1995; Arieli et al., 1996; Leopold and Maier, 2012). Functional magnetic resonance imaging (fMRI) (Ogawa et al., 1992) studies of the nature of this activity have revealed multiple spatial patterns of temporally correlated signal fluctuation that cover much of the brain, and often align with the established systems that support specialized brain functions (Biswal et al., 1995; Cordes et al., 2000; Hampson et al., 2002; Greicius et al., 2003; Fox et al., 2006). Based on this, it has been hypothesized that these correlational patterns of fMRI activity (often called "resting-state networks," or RSNs) (Fox and Raichle, 2007) indirectly result from spontaneous electrical activity in a number of distinct, large scale, and function-specific neuronal networks (NNs). Interpretation of RSNs in terms of the NNs supporting the major brain functions is an important goal of current neuroimaging research (Fox and Raichle, 2007; Biswal et al., 2010; Zhang and Raichle, 2010).

To tackle this challenging problem, a number of methods have been applied to analyze resting-state fMRI signals, including "seed"-based correlation analysis (Biswal et al., 1995; Fox et al., 2006), clustering based on temporal characteristics (Cordes et al., 2002; Mezer et al., 2009) and spatial or temporal independent component analysis (ICA) (Kiviniemi et al., 2003; Beckmann et al., 2005; Smith et al., 2012). These methods generally make implicit or explicit assumptions about the source signals underlying spontaneous fMRI activity, including stationarity of interregional interactions, and/or a statistical independence. The extent to which these assumptions reflect the nature of NNs determines how accurately they are represented by RSNs.

Recent studies have provided evidence that spontaneous brain activity may be non-stationary (Chang and Glover, 2010; Allen et al., 2012; Hutchison et al., 2012; Rack-Gomer and Liu, 2012), and in fact may be dominated by brief instances of spontaneous co-activation of brain regions (Tagliazucchi et al., 2012; Liu and Duyn, 2013; Wu et al., 2013). This has inspired a novel analysis approach that temporally decomposes conventional RSNs into multiple co-activation patterns by selective averaging of single fMRI time frames (Liu and Duyn, 2013). Here we extend this approach and perform a comprehensive analysis of a publicly accessible fMRI database to extract 30 spatial patterns of spontaneous activity (termed co-activation patterns or CAPs) that are biologically plausible and show distinct differences from networks extracted with conventional methods.

# **METHODS**

#### **ANALYSIS APPROACH**

The proposed analysis approach is based on the notion that spontaneous activity may be dominated by brief activations and deactivations involving many (possibly overlapping) brain regions. It differs from point process analysis (PPA), which models activity underlying fMRI signals as point processes and then examines the conditional distribution of these processes given its occurrence at a specific seed region (Tagliazucchi et al., 2012; Wu et al., 2013), in that it is not specifically geared toward detecting neuronalavalanche type activity. Rather than selectively averaging time points of activity increases in a seed region, the method proposed here classifies and averages time points with similar spatial distributions of activity using the *k*-means clustering algorithm. It thus extends the seed-based approach presented in a previous study (Liu and Duyn, 2013) to a data-driven, whole-brain analysis.

Clustering is a procedure for classifying a set of objects into different groups such that within group differences are smaller than across group differences. An approach popular with fMRI studies is the *k*-means clustering method, which has previously been applied to classify brain voxels based either on their temporal dynamics (Cordes et al., 2002; Mezer et al., 2009) or correlation profiles (Anderson et al., 2010; Kelly et al., 2012). Here, in contrast, we apply *k*-means clustering to classify fMRI time points (fMRI image volumes) based on their *spatial* similarity.

After reformatting fMRI brain volumes into a set of *m*-dimensional vectors {*t*1*, t*2*,..., tn*} (where *m* = the number of brain voxels, based on a brain mask created from the 152-brain MNI template), *k*-means clustering is applied to partition the vectors into *k* clusters *R* = {*R*1*, R*2*,..., Rk*} (Gluck and Myers, 1997) such that the sum of within-cluster distances *J* (Equation 1) is minimized:

$$J = \sum\_{i=1}^{k} \sum\_{t\_j \in R\_i} d(t\_j, \mu\_i) \tag{1}$$

where *µ<sup>i</sup>* is the mean of fMRI volumes in *Ri*, and *d*(•) represents the distance between two vectors, which we define here to be 1 minus their Pearson's correlation coefficient.

#### **DATASET**

The resting-state fMRI dataset used in this study was obtained from the 1000 Functional Connectomes Project (FCP) (http://www.nitrc.org/projects/fcon\_1000/) (Biswal et al., 2010). It includes data from studies performed independently at three different sites (Baltimore, Berlin\_Margulies, and Cambridge\_Buckner) with approval from their respective ethics committees. Due to computational limitations, we focused our analysis on data from 102 subjects (mean age: 24.4 ± 6.6, range: 18-44; sex: 64 females) selected from all 3 sites. Detailed information regarding each dataset and the major MR acquisition parameters can be found at the FCP website.

#### **PRE-PROCESSING OF RESTING-STATE fMRI SIGNALS**

FCP analysis scripts (version 1.1-beta, available at http://www. nitrc.org/frs/shownotes.php?release\_id=938) (Biswal et al., 2010), which employs AFNI (Cox, 1996) and FSL (http://www*.* fmrib*.*ox*.*ac*.*uk/fsl/) (Smith et al., 2004) software packages, were used to pre-process the fMRI signals (with minor modifications, described below). The typical pre-processing steps for the analysis of resting state data were applied, including image coregistration to correct for head motion, spatial smoothing with a Gaussian kernel (FWHM = 4 mm), temporal filtering with a band-pass filter (0.005–0.1 Hz), and the removal of linear and quadratic temporal trends. Additionally, the time series of ROIs in the white matter and cerebrospinal fluid (CSF), 6 affine motion parameters, as well as the brain-averaged (global) signal, were used as nuisance variables to be regressed out from the data. Given that global signal regression (GSR) may introduce artificial anti-correlation between regions (Fox et al., 2009; Murphy et al., 2009; Saad et al., 2012) our analysis was also performed without the GSR step (Figure S1).

The fMRI data was first co-registered to the high-resolution anatomical (T1-weighted) images acquired from the same subject and then normalized to the 152-brain Montreal Neurological Institute (MNI) normalized space. Here, as departure from the original FCP scripts, the registration between the functional and anatomical images was implemented using the "align\_epi\_anat.py" program (Saad et al., 2009) in AFNI, which was found to provide a better registration in the superior-inferior direction. The pre-processed fMRI data were then resampled at the 3 <sup>×</sup> <sup>3</sup> <sup>×</sup> 3 mm<sup>3</sup> resolution of the MNI normalized brain space. Finally, for each voxel, the fMRI signal was temporally normalized by subtracting its mean and then dividing by its temporal standard deviation (*SD*).

#### **EXTRACTION OF CAPs**

The clustering was applied to all 13382 fMRI volumes acquired from all 102 participants. After clustering, the fMRI volumes (also referred to as "time frames") assigned to the same cluster were simply averaged, resulting in *k* maps that we define as CAPs. These CAPs were then normalized by the standard error (within cluster and across fMRI volumes) to generate *Z-*statistic maps, which quantify the degree of significance to which the CAP map values (for each voxel) deviate from zero. We also calculated three quantities for each CAP based on its raw map: (1) the occurrence rate, which was calculated by dividing the number of fMRI time frames belonging to a given CAP by the total number of time frames; (2) the within-cluster similarity, calculated as the average spatial correlation of all within-cluster volumes to their mean; and (3) the polarity, calculated as the sum of the mean of positive map values and that of negative map values, such that the resulting sign indicates whether the CAP is dominated by activation or de-activation.

To suppress the contribution of measurement noise, only the largest signal changes were considered for calculation of distances for the clustering procedure. This was achieved by applying a mask to the fMRI volumes that only admitted the 10% highest and 5% lowest signal values, and discarded regions with less than 6 inter-connected (in 3D) voxels. This masking was only performed for the clustering and not for the within-cluster averaging procedure for calculation of the grand-average CAPs. Although this masking procedure was not strictly required and did not affect the general observations in this study, it was found to result in slightly more specific CAP patterns (Figure S2).

The number of clusters *k* was set to 30 after comparing the outcomes of setting *k* equal to 20, 30, and 40. Although these three cases generated largely similar CAPs, clustering with *k* = 30 led to a few distinct CAPs that were not found with *k* = 20, while *k* = 40 led to several CAPs that were nearly indistinguishable. Thus, the choice of *k* = 30 was a compromise between extracting too many and too few distinct CAPs base on what was afforded by the data.

#### **OCCURRENCE RATE OF CAPs**

For each fMRI session, the number of fMRI time frames belonging to a specific CAP was divided by the total number of frames in order to quantify the occurrence rate of the CAP in a subject (only one session for each participant). A permutation test was applied to determine whether the occurrence rates of the CAPs were significantly different between male and female subjects. Specifically, the 102 subjects were randomly assigned to two groups with size of 38 and 64, respectively, without considering their gender, and the difference in CAP occurrence rate between the two groups was recorded. This process was then repeated 50,000 times to build up distributions of the between-group difference for each CAP. The probabilities (*p*-values) of seeing differences between the male and female groups were then calculated by comparing the actual observations to these distributions, and a Bonferroni correction was applied to correct for multiple comparisons.

Head motion was also quantified for the male and female groups based on two metrics (mean translation, mean rotation) calculated from the 6 affine motion parameters. The mean translation was defined as the average displacement between any two consecutive volumes (defined as the root-mean-square of the translation parameters), and the mean rotation was calculated in similar way using the rotation parameters (Van Dijk et al., 2012).

#### **CORRELATION MAPS**

For comparison, seed-based correlation maps were calculated by correlating fMRI signals from all brain voxels to those from four seed regions (6 <sup>×</sup> <sup>6</sup> <sup>×</sup> 6 mm<sup>3</sup> cubes) centered at the following locations (with coordinates given in MNI space): posterior cingulate cortex [PCC, (0, −53, 26)], medical prefrontal cortex [mPFC, (0, 52, -6)], left intraparietal cortex [IPS, (−24, −58, 52)], and motor cortex [(−36, −25, 57)], respectively (Van Dijk et al., 2010).

#### **SPATIAL ICs AND TEMPORAL FUNCTIONAL MODES (TFMs)**

As another methodological comparison, group-level spatial ICA with temporal concatenation was implemented with the MELODIC program in the FSL software (Beckmann et al., 2005). The number of components to be extracted was specified as 30, matching the number of the clusters specified for the clustering. The CAPs were also compared to 21 TFMs derived previously by temporal ICA (Smith et al., 2012), which are available at SumsDB (http://sumsdb.wustl.edu/sums/directory.do?id=8288032&dir\_na me=TFM\_PNAS).

For comparison, the CAPs and ICs/TFMs that have highest spatial correlation to each other among their own category were paired up and shown in **Figure 6** and Figure S7.

#### **RESULTS**

#### **ACTIVITY DISTRIBUTION AT SINGLE TIME FRAMES**

Exemplary activity time frames (T1-T13) from a resting-state fMRI scan are shown in **Figure 1**, together with a map derived by temporally correlating signals across the brain with that from a "seed" region in the PCC. While the correlation map highlights the PCC, medial frontal cortex (MF), and bilateral parietal cortices (LPC), all recognized as primary nodes of an RSN known as the "default mode" network (DMN) (Greicius et al., 2003), single time frames often show clearly deviating patterns. For example, while T6, T11, T12, and T13 resemble DMN, T1 and T7 show coactivations at the visual cortex; T3 and T9 have high signal level specifically at the sensorimotor and insular cortex; T2, T4, T5, and T8 cover different sub-sections of the intraparietal sulcus (IPS) as well as some frontal areas, including the frontal eye fields (FEF); at T10, no apparent co-activation pattern is seen. It is worth noting that some of the patterns include strong co-activation at small but very specific brain structures, e.g., the hippocampus (HI) and posterior parahippocampal gyrus (PHG) in T11 and the ventral lateral nucleus (VL) of the thalamus in T3 (white arrows in **Figure 1**). Although some of these differences may be manifestations of experimental noise, their spatial characteristics (e.g., bilaterality and anatomical specificity) are suggestive of a neuronal origin.

#### **SPONTANEOUS CO-ACTIVATION PATTERNS (CAPs)**

To characterize stable, recurring co-activation patterns in spontaneous activity, single fMRI time-frames (13382 frames total) were clustered into 30 groups based on their spatial characteristics, after which within-group averaging was performed to calculate canonical CAPs (see Methods). These 30 CAPs were ranked by their similarity values, defined as the average correlation between each fMRI time frame within a group and their mean (Figure S3). Many of these CAPs resembled RSNs extracted with conventional analysis, suggesting network activity in the DMN, sensory regions with or without DMN involvement (see examples in Figure S3).

In various brain regions, CAPs showed interesting differences from RSNs (**Figure 2**). For example, DMN CAPs 1 and 15 selectively involved the HI, CAP 7 selectively involved the posterior PHG, and CAP 13 involved both. This distinct involvement of HI and PHG in DMN is not observed with conventional analysis. In addition, CAP 1 showed an asymmetric activation at the superior portion of the left middle frontal gyrus (MFG) even though its general pattern is fairly symmetric between hemispheres; CAP 7 has very specific activation along the superior frontal gyrus (SFG), and its LPC components localize more posteriorly compared to the others; and CAP 15 shows clear co-activations at the caudate nucleus (CN). Moreover, co-activations at the PCC/Precuneus regions in CAPs 7 and 13 appear to be patterned differently from those in CAPs 1 and 15. All these DMN-related CAPs are also associated with strong de-activations in a set of "task-positive" regions (Fox et al., 2005), although to varying spatial extent.

A second group of CAPs showed de-activations in DMNrelated regions combined with strong co-activations in a set of "task-positive" regions (**Figure 3**). Their patterns appeared more distinct from one another as compared to the DMN-activated CAPs. While CAPs 2 and 3 are both related to the visual system, CAP 2 overlaps more on the high order visual area and FEF. CAPs 4 and 6 looks almost identical in some axial slices, but a close comparison reveals that CAP 4 covers the central opercular cortex (CO), parietal operculum (PO), insular, thalamus, and supplementary motor area (SMA), whereas CAP 6 involves more anterior regions, including the frontal operculum (FO) and paracingulate gyrus (PCG). Moreover, even though all these CAPs show activations in the IPS, they cover very different sub-sections. There are also CAPs showing highly lateralized patterns (Figure S4) or strong activation in the primary sensory regions (Figure S5).

Several CAPs showed focal activity in thalamic and cerebellar structures. For example, CAPs 2 and 3, both of which involve higher order visual and visual association areas, include spatially distinct thalamic nuclei, with the ones in CAP 2 located more superior, medial, and anterior than those in CAP 3 (**Figure 4A**). In both CAPs, this thalamic activity appears to occur within the pulvinar. In contrast, CAP 26, whose activity shows a preference

**FIGURE 1 | Spontaneous activity at single fMRI time frames is suggestive of varying network involvement.** Sample frames (T1-T13) from a representative fMRI scan show clear, distinct spatial patterns, some of which (red points) resemble the DMN pattern demonstrated by the PCC-based correlation map of this subject (right). The fMRI time course

represents signal from the PCC seed region (green square). Small structures indicated by white arrows are the hippocampus (HI) and posterior parahippocampal gyrus (PHG) in T11 and the ventral lateral nucleus (VL) in T3. The color bar represents normalized BOLD signal for the sample frames and correlation values for the seed-based correlation map.

the DMN pattern, but differ in detail. For example, CAP 1, 3, and 15 show specific co-activations at the hippocampus (HI), parahippocampal gyrus (PHG), and caudate nucleus (CN), respectively. highest spatial correlation with the PCC-CorrMap, bottom right). Both methods failed to extract multiple patterns of DMN seen in the CAPs.

another as compared to the DMN-activated CAPs. While CAPs 2 and 3 are both related to the visual system, CAP 2 overlaps more on the high order visual area and frontal eye fields (FEF). CAPs 4 and 6

for primary visual cortex, includes bilateral focal structures that are much more inferior, lateral, and anterior than the thalamic structures seen in CAPs 2 and 3. We tentatively attribute these focal structures to the lateral geniculate nuclei (LGN). In contrast to the vision-related CAPs, thalamic nuclei in sensorimotorrelated CAPs tend to show activity opposite to the cortical regions. Strong de-activations (or activations) are observed in the anterior and medial dorsal nuclei (AN/MDN, big arrows in **Figure 4B**), and in some regions surrounding posterior thalamus (small arrows), when the sensorimotor cortex shows activation (resp. de-activation) in CAP 19 (resp. CAP 8). These regions are much less distinct (for the AN/MDN) or even absent (for the thalamus surrounding areas) in maps derived by seed-based correlation or ICA (note: the display thresholds for these latter maps were adjusted, base on the whole brain, to include approximately the same spatial extent of significantly negative regions). It should also be noted that a pair of thalamic nuclei showing co-activation with cortical regions could also be clearly seen in CAP 19 when lowering the display threshold (**Figure 4C**). This pair of nuclei is attributed to the ventral posterolateral nuclei (VPL), located about 6 mm posterior to the ventral posteromedial nucleus (VL) shown in CAP 23 (**Figure 4C**).

The 30 CAPs are available at our website (http:// amri.ninds.nih.gov/pub/xiao/CAPs\_30\_2mm.nii.gz).

though all these CAPs show activations in the IPS, they cover very

#### **OCCURRENCE RATE OF CAPs**

different sub-sections.

An interesting aspect of the proposed analysis approach is that it not only provides spatial maps of co-activating brain regions, but also allows extraction of incidence rates of spatially distinct co-activations, information that is not explicitly available with conventional analysis methods. This incidence or occurrence rate of CAPs may facilitate distinction between subject populations. To illustrate this, we examined the possibility of distinguishing between spontaneous brain activity in males (*n* = 38) and females (*n* = 64) based on differences in occurrence rates of all 30 CAPs. For this purpose, the occurrence rate of a CAP was determined from the fraction of total volumes that were classified into the CAP's associated cluster.

CAP 23, which primarily covered sensorimotor areas corresponding to head regions, was found to occur more frequently (*p <* 0*.*01, Bonferroni corrected, permutation test) in the male members than in the female members (**Figure 5**). However, the males (mean motion: 0.048 ± 0.021 mm) had a little more (*p* = 0*.*038, 2-sample *t*-test) head motion than the females (mean

motion: 0.040 ± 0.018 mm), which may have affected this result (Van Dijk et al., 2012). To exclude this confounding factor, the same comparison was repeated between 30 males and 39 females whose mean motions exceeded 0.03 mm, which resulted in subgroups with insignificant difference in their mean motion (*p* = 0*.*25) (we were unable to generate subgroups with similar member counts based on upper threshold for motion). These two subgroups continued to show a significant difference in occurrence rates of CAP 23 (Figure S6).

#### **COMPARISON TO OTHER DATA-DRIVEN APPROACHES**

A more comprehensive comparison of CAPs with independent components (ICs) derived from the same dataset is presented in Figure S7 and **Figures 6A,B**. ICs typically cover a small but specific set of brain areas with very high z-statistics. As a result of the spatial independence criterion, there is little overlap of the regions having high z statistics in different ICs (Figure S7), resulting in a spatial correlation very close to zero (**Figure 6A**). In contrast, several CAPs, particularly those related to DMN or task-positive regions (Figure S3, CAPs 1,7,12,15, and CAPs 2,3,4,6,9, respectively), cover relatively large brain areas and show substantial spatial correlations with one another (**Figure 6A**). The CAPs in this category do not have clear correspondence to the patterns seen in the 30 ICs. In contrast, the CAPs covering small but specific regions, e.g., those related to primary sensory systems, tended to more closely resemble ICs (Figure S7, CAPs 16,19,23,26,29). Judging from their mutual spatial crosscorrelation, at least a third of the CAPs did not pair up with any of the ICs (**Figure 6A**). Another noticeable difference between CAPs and ICs is asymmetric tails observed for the distribution of ICA map statistics (**Figure 6B**), which corresponds to the lack of significantly negative values in the ICs (Figure S7).

Recently, temporal ICA was applied to the components generated from spatial ICA in order to generate a set of temporal functional modes (TFMs) (Smith et al., 2012). Similarity between TFMs and CAPs may be expected, since temporal independence between TFMs is, to some degree, similar to the exclusive relation

bars represent the standard error across participants. The asterisk and light blue shadow indicates CAPs showing a significant difference in their occurrence (*p <* 0*.*01 with Bonferroni correction, permutation test) between these two groups.

of a time point to a specific CAP. To investigate this, we performed a spatial similarity analysis comparing the 30 CAPs with 21 TFMs derived in Smith et al. (2012). Although there were several CAP— TFM pairs with high similarity (CAP 2 and TFM 8, CAP 12 and TFM 19, and CAP 26 and TFM 2), most CAPs did not have a matching TFM (**Figures 6A,C**).

# **DISCUSSION**

In this report, we demonstrated a method to extract brain functional information by identifying regions that spontaneously co-activate during resting conditions. The method is based on the notion that co-activation of functionally related brain regions may not be a continual, stationary phenomenon (is implicitly assumed in conventional analysis) but rather occur sporadically over the course of a few seconds. The method also extends its previous version of seed-based analysis targeting specific networks (Liu and Duyn, 2013) to a data-driven, whole-brain approach.

By applying this approach to a large dataset of resting state fMRI studies, we identified at least 30 reproducible, spatially distinct co-activation patterns (CAPs). These CAPs in some aspects resemble canonical RSNs and TFMs extracted with ICA, but also

**FIGURE 6 | Comparison between CAPs, ICs, and TFMs. (A)** Spatial cross correlation matrices indicating substantial differences between CAPs (middle) on one hand and ICs and TFMs on the other (top and bottom, respectively). Cross-modality correlations were sorted based on correlation strength. **(B)**

Distributions of map statistics (z-statistics) for CAPs (white), ICs (red), and TFMs (blue). IC statistics show a very non-Gaussian distribution with a elevated right tail, consistent with the independence assumption of ICA. **(C)** A selection of best-matched CAPs and TFMs, based on spatial similarity.

show distinct differences in various brain regions. These differences are partly attributed to methodological differences, for example the fact that the proposed method does not assume statistical independence between CAPs, as is the case for RSNs and TFMs extracted with ICA. Thus, the proposed method may lead to an alternative interpretation of the NNs underlying resting-state fMRI signals.

#### **DISTINCT CO-ACTIVATION PATTERNS AND DYNAMIC FUNCTIONAL CONNECTIVITY**

A series of recent studies has attempted to reveal dynamic changes in RSNs by examining the temporal variability in fMRI signal correlations (Chang and Glover, 2010; Allen et al., 2012; Hutchison et al., 2012; Rack-Gomer and Liu, 2012). Considerable variations in RSN characteristics were found even over the duration of a typical (several minute) resting scan, suggesting that the underlying NNs may dynamically assemble and disassemble over this time window. Such temporal variations may pose significant challenges for correlational approaches for the extraction of RSNs, as it may render them strongly dependent on the choice of analysis window. For example, the PCC-seeded correlation map within a 16.1-s window (only 7 time points) including T11 shows a DMN pattern with high correlations at the HI and the posterior PHG (Figure S8) because they co-activated at T11 (**Figure 1**); their functional connectivity to the PCC seed, however, nearly vanishes when the time window is shifted forward by 23 s to cover T12 instead. Likewise, the thalamic nucleus seen in T3 is only present in correlation maps seeded at the motor cortex for time windows including T3 (Figure S8). This dependence on position of the analysis window can be mitigated or avoided by increasing the window duration, which leads to more stable, reproducible RSN, but on the other hand can result in the loss of network information contained in single time frames.

#### **INFORMATION IN CAPs**

Using the method proposed in this study, the analysis window was effectively reduced to include only a single time point, and sensitivity was subsequently improved through selectively averaging the fMRI volumes with similar patterns. By averaging selectively rather than continually (as implicitly occurs with correlation analysis), one may extract finer detail regarding spontaneous coactivations of multiple brain regions. For example, correlation analysis with a seed in the PCC and a typical analysis window of a few minutes generally reveals a functional connection with HI and posterior PHG but little other information. In contrast, the multiple DMN-related CAPs extracted with the proposed method suggests that the two regions at times independently connect to the PCC. A possible explanation for these regions' differential involvement with DMN is their distinct roles in memory storage and retrieval (Gluck and Myers, 1997). Similarly, the varying involvement of sensory regions with the DMN apparent from CAPs (**Figure 3**) may signify that the brain may not be simply organized into two anti-correlated NNs spanning "taskpositive" and "task-negative" regions, as concluded previously from correlation analysis (Fox et al., 2005).

Many CAPs included small structures in the thalamus and cerebellum that appeared to localize to specific anatomic subregions. The vision-related CAPs included thalamic nuclei that were well-separated and corresponded to established targets of functional connections with specific regions of the visual cortex that co-occurred in the CAPs. Similarly, the sensorimotor CAPs corresponding to the head (CAP 23) and lower body (CAP 19) regions showed specific co-activations at the VL and VPL, respectively, despite their limited anatomical separation of only a few millimeters. Anti-phase interaction was also observed between thalamic nuclei and sensorimotor cortex in CAPs 8 and 19. The AN/MDN in these CAPs has been shown to positively correlate with the alpha-band (8-13 Hz) electroencephalography (EEG) power, which is, in turn, negatively correlated with multiple sensory regions, including the sensorimotor cortex (Liu et al., 2012). These thalamic regions have non-specific connections to the cortex and may participate in the process of alertness and arousal (Van Der Werf et al., 2002). The second set of thalamic regions showing negative correlation to the sensorimotor cortex was around the LGN. Lowering the display threshold exposed multiple diffuse areas encompassing the posterior thalamus, which we suspects to be the thalamic reticular nucleus (TRN), partly because the anti-phase activity with sensorimotor cortex is, to some extent, consistent with the established inhibitory influence of TRN over other thalamic nuclei.

A few CAPs included isolated cortical structures that were distant from regions of major activity. For example, CAP 16, which includes areas associated with peripheral vision, showed small co-activation in the SMA, motor cortex, and medial IPS. This is plausible considering the critical role of the medial IPS in visuomotor coordinate transformation (Grefkes et al., 2004). These subtle aspects of CAPs are generally not captured with ICA or correlation analysis, indicating potential advantages of the proposed analysis approach.

Since hemodynamic response function (HRF) has been demonstrated to vary spatially (Wu et al., 2013), a potential concern is that different CAPs may actually represent the same neuronal co-activation event at different hemodynamic delays. This is, however, unlikely for the following two reasons. First, the fact that CAPs, particularly those covering similar or close cortical regions, show specific co-activations in distinct thalamic and/or cerebellar structures and that many of these CAP patterns are consistent with known organization of the brain argue strongly against their origin being attributable to vascular effects (e.g., hemodynamic delay differences between regions). Secondly, the analysis of temporal precedence of CAPs failed to find any pair of CAPs occurring in a specific order (Figure S9), which is also inconsistent with the confounding effect of the HRF delays. Conversely, co-activation in the same CAP does not guarantee the exact synchronization of neural activity at corresponding regions. Therefore, the "co-activation" in CAP refers to synchronization of fMRI signals rather than that of the underlying neural activity.

We found both co-activation and co-deactivation CAPs for some brain regions (e.g., CAP 8 and CAP 19), which are actually averages around fMRI signal peaks and troughs, respectively. Although these two types of CAPs often show very similar spatial patterns with reversed sign, a close inspection revealed that the co-activation at the fMRI peaks is significantly more synchronized than the co-deactivation at the troughs (Figure S10). The implication of this observation remains unclear, but the peaks and troughs are clearly corresponding to distinct cortical states.

An attractive aspect of the proposed analysis is that it provides a simple quantifiable measure of CAP occurrence rate that facilitates comparison between subject populations. As example of this, we demonstrated that CAP 23, which overlaps the sensorimotor cortex, occurs more frequently in males than in females. Although far from providing a physiological explanation, this finding is consistent with a previous study reporting sex differences in the fluctuation amplitude of resting-state fMRI signals in similar brain regions (Biswal et al., 2010). We also make publicly available the 30 CAPs derived in our study, which can be directly used as templates for future studies intending to examine and compare their occurrence rates in different groups or under different conditions.

#### **COMPARISON WITH CONVENTIONAL METHODS**

As mentioned above, the extraction of CAPs based on single-time frame analysis may allow one to better capture temporally localized spontaneous co-activations as compared to conventional correlation analysis. The method presented here resembles spatial ICA in the sense that both are spatial-domain methods that regard fMRI time frames as the basic units of analysis. However, ICA assumes that the fMRI volumes comprise weighted combinations of a set of statistically ICs, the best fit of which can be found by maximizing the independence of the latter. The independence assumption yields ICs that typically cover a relatively small extent of the brain with very high statistical scores (Figure S7), because such a pattern has high non-Gaussianity (**Figure 6B**), an important factor underlying ICA based signal separation. At the same time, independence maximization yields ICs with minimal pairwise spatial correlations, as shown in **Figure 6A**. Together, these two features suggest that the ICs present a spatial parcellation of the brain rather than distinct states of functional connectivity. In addition, ICs may exhibit considerable pairwise *temporal* correlation as exemplified by the often negative correlation observed between the ICs covering task-positive and task-negative regions.

To overcome these limitations and gain more knowledge about the brain's spontaneous co-activations, temporal ICA has been further applied to spatial ICs to recombine them into a set of TFMs (Smith et al., 2012). The TFMs are perhaps more closely related to the CAPs, since the concept of temporal independence is similar to the exclusive presence of a given CAP at any given time. However, a direct comparison between TFMs and CAPs indicated substantial differences (**Figure 6**). This may not be surprising, given the rather different methodology and assumptions used to generate these network measures.

A practical advantage of the proposed method is that it involves few assumptions and transformations of the data. The classification procedure (clustering) does not perform any transformation on the data, and the CAPs are simply averages of time frames classified into groups based on spatial activity patterns. Therefore, they are easy to interpret and reflect the underlying brain activity in a rather direct way.

#### **NEURAL ORIGIN OF CAPs**

Since the co-activation patterns are clear even in single fMRI volumes, they likely reflect large-scale neural activity occurring within brief time periods. One candidate is large-scale neuronal avalanching activity, defined as spontaneous activity initiating at a specific location in the form of a brief burst, and taking with it, like an avalanche, other connected regions that are near activation threshold (Beggs and Plenz, 2003; Tagliazucchi et al., 2012). The previous analysis on spatiotemporal dynamics of spontaneous fMRI signals confirmed its scale-free property, consistent with avalanche types of activity at smaller time-scales (Tagliazucchi et al., 2012). However, it is also possible that other types of neuronal activity underlie such brief co-activations of the brain. For example, since the dataset we analyzed here was acquired during wakefulness, it is possible that neural activity associated certain conscious processes, e.g., mind wandering, may also contribute to spontaneous co-activations of the brain. The analysis of fMRI signals acquired under brain states with reduced consciousness would help to partially clarify these issues. Nevertheless, uncovering the precise neuronal origin and functional relevance of the CAPs will likely require acquisition of electrophysiological data concurrently acquired with the fMRI.

#### **TECHNICAL LIMITATIONS**

A challenge for clustering analysis is selection of *k*, i.e., the number of CAPs to be extracted from the data. The current value of 30 was chosen somewhat arbitrarily from a comparison of results obtained with *k* = 20, 30, and 40. We have also attempted to select *k* by evaluating the corresponding cluster structures with more objective indices, including the Silhouette, Calinski-Harabasz, Davies-Bouldin methods, which, however, yielded inconsistent recommendations that also tend to underestimate the number of CAPs present in the data. This may be due to the two following reasons: first, the distance (dissimilarity) among the CAPs is likely to have a very skewed distribution, with a portion of the CAPs being much closer to one another than to the others (**Figure 6A**), which increases the difficulty in finding a clear division. Secondly, the noisy fMRI volumes mentioned above will further blur the boundaries between different CAPs. Nevertheless, the CAPs in the current dataset were found to be rather insensitive to this parameter, as the majority of the CAPs shown for the case of *k* = 30 were also observed in the cases of *k* = 20 and *k* = 40.

The computational load of the *k*-means clustering increases quickly with the number of fMRI volumes (time points). Therefore, the application of the proposed method to fMRI datasets with very high sampling rate, e.g., those acquired with the newly developed multi-band technique, would either be limited to a relatively small population or have to require more computational resources and/or more efficient computational algorithms. This could be a challenge for future studies attempting to utilize the ultra-high temporal resolution of fMRI data for special research purposes.

#### **ACKNOWLEDGMENTS**

This research was supported (in part) by the Intramural Research Program of the NIH, NINDS.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnsys.2013. 00101/abstract


and anesthetized macaques*. Hum. Brain Mapp.* 34, 2154–2177. doi: 10.1002/ hbm.22058


networks from resting state fMRI data. *Med. Image Anal.* 17, 365–374. doi: 10.1016/j.media.2013.01.003

Zhang, D., and Raichle, M. E. (2010). Disease and the brain's dark energy. *Nat. Rev. Neurol.* 6, 15–28. doi: 10.1038/nrneurol.2009.198

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 10 July 2013; accepted: 15 November 2013; published online: December 2013. 04*

*Citation: Liu X, Chang C and Duyn JH (2013) Decomposition of spontaneous brain activity into distinct fMRI co-activation patterns. Front. Syst. Neurosci. 7:101. doi: 10.3389/fnsys.2013.00101*

*This article was submitted to the journal Frontiers in Systems Neuroscience.*

*Copyright © 2013 Liu, Chang and Duyn. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Cortical Variability and Challenges for Modeling Approaches

Emili Balaguer-Ballester 1, 2 \*

<sup>1</sup> Department of Computing and Informatics, Faculty of Science and Technology, Bournemouth University, Bournemouth, UK, <sup>2</sup> Bernstein Center for Computational Neuroscience, Medical Faculty Mannheim and Heidelberg University, Mannheim, Germany

#### Keywords: metastability, excitation-inhibition balance, cortical variability, neuronal variability, recurrent networks, trial to trial variability

The functional role of the observed neuronal variability (the disparity in neural responses across multiple instances of the same experiment) is again receiving close attention in Computational and Systems Neuroscience (e.g., Durstewitz et al., 2010; Moreno-Bote et al., 2011; Oram, 2011; Beck et al., 2012; Churchland and Abbott, 2012; Brunton et al., 2013; Masquelier, 2013; Mattia et al., 2013; Balaguer-Ballester et al., 2014; Renart and Machens, 2014; Bujan et al., 2015; Lin et al., 2015; Pachitariu et al., 2015; Arandia-Romero et al., 2016; Doiron et al., 2016; McDonnell et al., 2016). Special consideration is currently given to understanding how spiking (Bujan et al., 2015; Deneve and Machens, 2016; Doiron et al., 2016; Hartmann et al., 2016; Landau et al., 2016) and phenomenological (Goris et al., 2014; Lin et al., 2015; Mochol et al., 2015; Arandia-Romero et al., 2016; Doiron et al., 2016) models account for the wide range of classical and new phenomena associated with trial-to-trial uncorrelated activity.

#### Edited by:

Gilad Silberberg, Karolinska Institutet, Sweden

#### Reviewed by:

Maurizio Mattia, Istituto Superiore di Sanità, Italy Arvind Kumar, Royal Institute of Technology, Sweden Iñigo Arandia-Romero, Pompeu Fabra University, Spain

#### \*Correspondence:

Emili Balaguer-Ballester eb-ballester@bournemouth.ac.uk

> Received: 18 October 2016 Accepted: 06 March 2017 Published: 04 April 2017

#### Citation:

Balaguer-Ballester E (2017) Cortical Variability and Challenges for Modeling Approaches. Front. Syst. Neurosci. 11:15. doi: 10.3389/fnsys.2017.00015

Specifically, it has often been proposed that a network state characterized by largely asynchronous spike times whilst maintaining slow oscillations in the firing-rates, may represent the default spontaneous cortical mode (e.g., Sanchez-Vives and Mattia, 2014; Deneve and Machens, 2016; Sancristobal et al., 2016); and similar states could also underlie observed stimulus-driven variability in rate (Litwin-Kumar and Doiron, 2012; Deneve and Machens, 2016; Hartmann et al., 2016). However, the way in which such a computationally advantageous network state for neural coding is achieved can differ substantially between modeling approaches; this challenge will be the focus of this manuscript.

# PREDICTABLE COMPONENTS OF NEURONAL VARIABILITY

The view that the intrinsic stochasticity of single cell activity is the major source of variability has been questioned multiple times over the last decades by modeling (van Vreeswijk and Sompolinsky, 1996; Amit and Brunel, 1997; Shadlen and Newsome, 1998; Deneve et al., 2001; Stein et al., 2005; Faisal et al., 2008; Renart et al., 2010; Rabinovich and Varona, 2011; Masquelier, 2013; Stiefel et al., 2013; Rabinovich et al., 2014; Deneve and Machens, 2016; Hartmann et al., 2016) and empirical studies (e.g., Bryant and Segundo, 1976; Mainen and Sejnowski, 1995; Britten et al., 1996; Stein et al., 2005). It is well known that essentially deterministic networks of balanced excitation and inhibition are able to generate a weakly correlated, often chaotic attractor state which presents Poissonian statistical properties like the observed activity (van Vreeswijk and Sompolinsky, 1996; Amit and Brunel, 1997; Shadlen and Newsome, 1998; Sussillo and Abbott, 2009; Litwin-Kumar and Doiron, 2012). However, such a chaotic state is a non-mandatory modeling choice: recently, a range of models has shown that part of the observed variability may also be explained by a different class of deterministic processes (Beck et al., 2012; Mattia et al., 2013; Renart and Machens, 2014; Bujan et al., 2015; Abbott et al., 2016; Deneve and Machens, 2016; Doiron et al., 2016; Gillary and Niebur, 2016; Hartmann et al., 2016) such as the lack of specificity in top-down processing of cognitively complex tasks (Beck et al., 2012).

At the same time, empirical studies found mounting evidence of deterministic patterns for some of the trial to trial variability. For example, a range of indexes (Shadlen and Newsome, 1998; Churchland and Abbott, 2012; Marcos et al., 2013) suggest that variance is systematically reduced at the stimulus onset (Churchland et al., 2010); and in general shows a predictable trend during different events of the task (Churchland et al., 2006, 2010; Churchland and Abbott, 2012; Ledberg et al., 2012; Renart and Machens, 2014). Thus, there seems to be an increasing consensus in that at least part of the trial to trial variability shows a deterministic pattern which may play a functional role; and hence cannot be simply neglected (Balaguer-Ballester et al., 2011, 2014; Masquelier, 2013; Ecker et al., 2014; Goris et al., 2014; Renart and Machens, 2014; Lin et al., 2015; Schölvinck et al., 2015; Arandia-Romero et al., 2016; Hartmann et al., 2016).

Nevertheless, despite such recent advances, the mapping between the cognitive state and variability is still challenging. For instance, on the one hand, correlated rate variability between pairs of neurons is often reduced by top-down attentional processes (e.g., Cohen and Maunsell, 2009; Mitchell et al., 2009; Cohen and Kohn, 2011; Doiron et al., 2016). On the other hand, the opposite can be observed when attention is highly variable across trials (Roelfsema et al., 2004; Renart and Machens, 2014; Ruff and Cohen, 2014); and such noise correlation analyses show a variety of mixed results (Cohen and Kohn, 2011; Eyherabide and Samengo, 2013; Moreno-Bote et al., 2014; Ruff and Cohen, 2014, 2016; Doiron et al., 2016).

Importantly, compelling evidence suggests that a substantial portion of the spontaneous and evoked total and shared variability is attributable to global fluctuations (Ecker et al., 2014, 2016; Goris et al., 2014; Mochol et al., 2015; Pachitariu et al., 2015; Schölvinck et al., 2015; Arandia-Romero et al., 2016); and this has direct implications in neural coding in visual (Lin et al., 2015; Arandia-Romero et al., 2016; Ecker et al., 2016) and in auditory areas (Mochol et al., 2015; Pachitariu et al., 2015). For instance, high population activity in monkey V1 increases the information that a subset of neuronal ensembles carry about stimulus orientation, only the ones that show a strong multiplicative modulation. In contrast, the stimulus-decoding information of such multiplicative ensembles plummets for low global activity states; whilst information increases in the group additively-modulated neurons in the population (Arandia-Romero et al., 2016).

Global modulations could either stem from the default up/down state of ongoing activity (Mochol et al., 2015) or from fluctuations within a single state (Arandia-Romero et al., 2016). When controlled for this global co-modulations, noise correlations are often negligible (Renart et al., 2010), but not always (Pachitariu et al., 2015; Arandia-Romero et al., 2016). Moreover, stimulus-driven input statistics can also have a strong contribution to the observed evoked variability in parallel to the global network state (Oram, 2011; Bujan et al., 2015; Pachitariu et al., 2015; Doiron et al., 2016; Landau et al., 2016) and explain noise correlations dynamics (Bujan et al., 2015).

This complex variety of results has been recently analyzed using a range of phenomenological and spiking models. These recent modeling efforts aim to pin down when precisely during the course of the trial (Moreno-Bote et al., 2014; Bujan et al., 2015; Doiron et al., 2016) and in which specific network state (Arandia-Romero et al., 2016) noise correlations are informative or deleterious for neural coding (Ecker et al., 2014, 2016; Moreno-Bote et al., 2014; Lin et al., 2015; Pachitariu et al., 2015; Schölvinck et al., 2015; Arandia-Romero et al., 2016; Doiron et al., 2016).

# DIVERSITY OF THEORETICAL APPROACHES

The consensus on the network origin of a substantial part of cortical variability led to the development of a multitude of models for explaining the underlying neuronal mechanisms of the asynchronous state (e.g., Boerlin et al., 2013; Deco et al., 2014; Ostojic, 2014; Barral and Reyes, 2016; Hartmann et al., 2016; Rosenbaum et al., 2017). A linking theme in these approaches is the crucial contribution of fast inhibition in recurrent networks; which is negatively correlated with excitation and strong enough to counterbalance it to different degrees (Renart et al., 2010; Deneve and Machens, 2016).

FIGURE 1 | Three dimensions in models of neuronal variability. The diamond shows an example of a spiking model with random connectivity and a tight Excitation/Inhibition (E/I) balance as in Renart et al., 2010; the circle represents a substantially different modeling choice such as in Hartmann et al., 2016. A range of modeling approaches typically fall between these two examples (triangles); such as semi-structured connectivity architectures which modulate the E/I balanced dynamics in realistic networks (Litwin-Kumar and Doiron, 2012; Landau et al., 2016).

This scenario is currently the subject of a lively debate; and a variety of processing architectures of spiking units have been developed to explain the observed variability phenomena from different perspectives. It has recently been proposed that a much tighter synchronization between excitation and inhibition than considered so far, at the spike level, has an even stronger experimental support and would enable the network to operate optimally by reducing the minimum coding error (Renart et al., 2010; Boerlin et al., 2013; Abbott et al., 2016; Deneve and Machens, 2016). The precise way in which the asynchronous state is achieved however, is not unique. For example, connectivity weights are specifically learnt in the design termed spike-coding network (Boerlin et al., 2013; Schwemmer et al., 2015; Abbott et al., 2016; Deneve and Machens, 2016); whilst connectivity is clustered in Litwin-Kumar and Doiron (2012), shaped by plasticity in Vogels et al. (2011) and Landau et al. (2016) and much less structured in many other dense (Renart et al., 2010; Abbott et al., 2016) or sparse (Ostojic, 2014) networks.

In contrast, other recent approaches rely on non-closely balanced excitatory-inhibitory dynamics in networks of simplified units. For instance, Hartmann et al. (2016) proposed a fully deterministic approach to describe spontaneous and stimulus evoked variability; consisting of an architecture of schematic noise-free units. In this approach, excitatory connectivity is specifically set by plasticity and homeostasis; and is not necessarily balanced. In Deco et al. (2014), the spontaneous state also stems from not necessarily tightly balanced architectures, where elements are field equations derived from spiking units with background input noise. In this and related models, connectivity is also set; but in the latter case local inhibition is regulated by a different homeostatic control. In addition, it has been recently shown that most of the evoked variability could be accounted for by essentially feedforward architectures (Bujan et al., 2015; Doiron et al., 2016).

Asymmetry and slightly unbalanced configurations are also considered to promote the so called metastable state (Mattia et al., 2013; Tognoli and Kelso, 2014a,b; Deco and Kringelbach, 2016), in which high dimensional ensembles flexibly re-organize, synchronize and disengage, possibly by changing their role in a high-dimensional setting (Lapish et al., 2015; Fusi et al., 2016). Such state also exhibits advantageous computational properties (Hellyer et al., 2015; Deco and Kringelbach, 2016). In an instantiation of such ideas, essentially deterministic structures of simplified units generate itinerancy through robust transient states. These states enable the model to process cognitive entities without the compelling need for attractors (Rabinovich et al., 2008, 2014; Varona and Rabinovich, 2016). Moreover, combinations of attractor-based and transient-based

# REFERENCES

Abbott, L. F., DePasquale, B., and Memmesheimer, R.-M. M. (2016). Building functional networks of spiking model neurons. Nat. Neurosci. 19, 350–355. doi: 10.1038/nn.4241

computations could also underlie motor plans (Mattia et al., 2013). Other state-dependent computational ingredients such as neuromodulation could also play a major role in shaping the observed variability (Mattia et al., 2013; Lapish et al., 2015; Doiron et al., 2016).

# FUTURE CHALLENGES FOR MODELS

This non-exhaustive summary of few recent examples suggests the availability of a plethora of recurrent and feedforward network models for understanding the source of variability during cognitive processing and in the resting state. These configurations often differ at least in the level of detail of the computational units, in the connectivity structure and in the degree of balance between excitation and inhibition (**Figure 1**).

This challenging scenario perhaps compels to the development of novel approaches for probing the networks in order to identify the suitable architecture or architectures for each specific cognitive process and cortical area. However, the question remains how to effectively dissect a recurrent network, beyond the linearization of the network dynamics, in order to investigate the components originating the asynchronous state (Sussillo, 2014). Recently, Doiron et al. (2016) proposed a framework to identify the physiological processes underlying decorrelation in feedforward circuits by analysing statedependent correlations in different time windows. However, applying this approach is more problematic in recurrent circuits when coupling is not weak and is highly nonlinear; as is often the case in models.

Hence, inferring which level of detail and architecture are mostly probably responsible for the neuronal variability phenomena is perhaps one of the major challenges for the next years, which possibly requires the development of novel theoretical tools for scrutinizing network behavior.

# AUTHOR CONTRIBUTIONS

The author conceived and wrote the manuscript.

# FUNDING

This paper has been funded by Bournemouth University. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

# ACKNOWLEDGMENTS

The author wants to thank very much the three reviewers for their valuable comments.


population activity affects encoded information. Neuron 89, 1305–1316. doi: 10.1016/j.neuron.2016.01.044


Oram, M. W. (2011). Visual stimulation decorrelates neuronal activity. J. Neurophysiol. 105, 942–957. doi: 10.1152/jn.00711.2009


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2017 Balaguer-Ballester. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.