# INFLUENCE OF INTER- AND INTRA-SYNAPTIC FACTORS ON INFORMATION PROCESSING IN THE BRAIN

EDITED BY : Vito Di Maio and Jean-Marie C. Bouteiller PUBLISHED IN : Frontiers in Computational Neuroscience, Frontiers in Synaptic Neuroscience and Frontiers in Neural Circuits

#### Frontiers Copyright Statement

© Copyright 2007-2019 Frontiers Media SA. All rights reserved. All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply.

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use. ISSN 1664-8714 ISBN 978-2-88963-073-8 DOI 10.3389/978-2-88963-073-8

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# INFLUENCE OF INTER- AND INTRA-SYNAPTIC FACTORS ON INFORMATION PROCESSING IN THE BRAIN

Topic Editors:

Vito Di Maio, Institute of Applied Sciences and Intelligent Systems (ISASI) of CNR, Italy

Jean-Marie C. Bouteiller, University of Southern California, United States

Image: Jean-Marie Bouteiller

Any brain activity relies on the interaction of thousands of neurons, each of which integrating signals from thousands of synapses. While neurons are undoubtedly the building blocks of the brain, synapses constitute the main loci of information transfer that lead to the emergence of neuronal code. Investigating synaptic transmission constitutes a multi-faceted challenge that brings together a large number of techniques and expertise ranging from experimental to computational approaches, bringing together paradigms spanning from molecular to neural network level.

In this book, we have collected a series of articles that present foundational work aimed at shedding much-needed light on brain information processing, synaptic transmission and neural code formation. Some articles present analyses of regulatory mechanisms underlying neural code formation and its elaboration at the molecular level, while others use computational and modelling approaches to investigate, at synaptic, neuronal and inter-neuronal level, how the different mechanisms involved in information processing interact to generate effects like long-term potentiation (LTP), which constitutes the cellular basis of learning and memory. This collection, although not exhaustive, aims to present a framework of the most used investigational paradigms and showcase results that may, in turn, generate novel hypotheses and ideas for further studies and investigations.

Citation: Di Maio, V., Bouteiller, J.-M. C., eds. (2019). Influence of Inter- and Intra-Synaptic Factors on Information Processing in the Brain. Lausanne: Frontiers Media. doi: 10.3389/978-2-88963-073-8

# Table of Contents

*05 Editorial: Influence of Inter- and Intra-Synaptic Factors on Information Processing in the Brain*

Vito Di Maio and Jean-Marie C. Bouteiller


Jean-Marie Bouteiller and Theodore W. Berger

*54 Dopamine Receptor Activation is Required for GABAergic Spike Timing-Dependent Plasticity in Response to Complex Spike Pairing in the Ventral Tegmental Area*

Ludovic D. Langlois, Matthieu Dacher and Fereshteh S. Nugent


Yan Chastagnier, Enora Moutin, Anne-Laure Hemonnot and Julie Perroy


Denis A. Adamchik, Valery V. Matrosov and Victor B. Kazantsev


# Editorial: Influence of Inter- and Intra-Synaptic Factors on Information Processing in the Brain

Vito Di Maio<sup>1</sup> \* and Jean-Marie C. Bouteiller <sup>2</sup> \*

*1 Istituto di Scienze Applicate e Sistemi Intelligenti (ISASI) del CNR, Pozzuoli, Italy, <sup>2</sup> Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, United States*

Keywords: synaptic transmission, dendritic integration, synaptic plasticity, neural modeling, neural network, glutamatergic, GABAergic

**Editorial on the Research Topic**

#### **Influence of Inter-and Intra-Synaptic Factors on Information Processing in the Brain**

The main function of the brain is to process and integrate information coming from the environment and from other parts of the body and produce appropriate responses.This information processing capability is mimicked in neurons in which dendritic arborization integrates information from thousands of synapses, both excitatory and inhibitory. The main activity of a neuron then consists in receiving synaptic inputs and integrate them to produce spike sequences (neural code), which ultimately form the presynaptic inputs to other neurons. Evidently, synaptic transmission represents an essential building block of information processing in the brain.

Synapses show large variability in the amplitude, time duration, and time-dependent probability of their produced output as a function of a presynaptic stimulus. Partially, this variability can be assumed to be of stochastic origin (e.g., position of the vesicles at the presynaptic side), but the larger part is due to the fine regulation of synaptic components, e.g., the activity-dependent regulation of the number of postsynaptic receptors (through long-term potentiation and long-term depression), their sub-unit composition and the biophysical properties of the neuronal area where they are located.

These regulatory mechanisms determine the intensity, frequency, ability, and quality of the information transmitted; they, in addition to the biophysical properties of both pre and post-synaptic neurons are responsible for shaping the whole neural activity.

Unsurprisingly, these mechanisms and their effects on neuronal activity in both physiological and pathological cases have been intensely studied. Yet a lot remains to be learned and achieving a better understanding of the mechanisms involved is still of paramount importance.

The present research topic aims to collect papers by authors who, using approaches of both experimental and computational nature, strengthen our understanding of synaptic function and its role in information processing. This collection provides a non-exhaustive yet multiscale perspective on synapses, ranging from the molecular contribution of proteins on the regulation of synaptic transmission, to the study of the effects of synaptic activity on network-level activity.

A molecular switch involving the protein CaMKII in short period plasticity is proposed by Clarke, who analyses the bistable properties of this molecule driven by Ca2<sup>+</sup> cytosolic transients using a computational approach.

To study stochastic short-term plasticity, a method based on hidden Markov models (HMMs) is proposed by Mokhtari et al.

Edited and reviewed by: *Si Wu, Peking University, China*

#### \*Correspondence:

*Vito Di Maio vito.dimaio@cnr.it Jean-Marie C. Bouteiller jbouteil@usc.edu*

Received: *27 March 2019* Accepted: *02 May 2019* Published: *21 May 2019*

#### Citation:

*Di Maio V and Bouteiller J-MC (2019) Editorial: Influence of Inter- and Intra-Synaptic Factors on Information Processing in the Brain. Front. Comput. Neurosci. 13:32. doi: 10.3389/fncom.2019.00032*

Guerrier and Holcman present a review of the intricate interactions that take place in the presynaptic terminal leading to the release of neurotransmitters; the presynaptic sensors for Ca2<sup>+</sup> are considered for their role in the probability of vesicle release and the contribution to plasticity phenomena.

All synaptic inputs are not equal and they strongly depend on the location of the synapses on the dendritic tree. Felton et al. present a study of postsynaptic integration and the interactions between multiple functional zones. They use stochastic resonance models to perform their analysis and demonstrate the presence of meaningful interactions between these different functional zones.

GABA is known to mediate intercellular communications by participating in both "wiring" and "volume" transmission. The "wiring" action of GABA takes place via synaptic transmission by activating the postsynaptic (phasic) GABAA-receptors. The "volume" transmission is carried out by "overspilled" ambient GABA which regulates neuronal excitability by creating extra transmembrane current through extrasynaptic (tonic) GABAAreceptors. In their contribution, Adamchik et al. propose a simple computational model that simulates the effects of nonstationary, activity dependent GABA upon population dynamics of interneurons and demonstrate that this effect leads to relaxation oscillations.

Ferguson and Gao present a study of the regulatory effects of inhibitory synapses on the activity of prefrontal cortical circuits neurones. They outline the role of inhibition in regulating the excitatory activity of glutamatergic neurons showing how lack of balance between excitatory and inhibitory synapses can be the base of specific pathologies.

Spike-timing dependent plasticity (STDP) is considered one of the major processes underlying memory storage and recall. Langlois et al. show the importance of dopaminergic synapses on the regulation of the GABAergic STDP, which plays a critical role in the circuitry between the deep nuclei of the brain and the cortex.

Other mechanisms have also been shown to contribute to modulation of STDP; Foncelle et al. propose a review of the different mechanisms reported to affect this modulation.

Based on a theoretical model, Millán et al. show that the pruning (and formation) of synapses alters the energy landscape of an associative network such that the neural system becomes able to track several memories or attractors by oscillationlike dynamics. They argue that this oscillation is induced by destabilization of the current attractor.

Calcium dynamics has been shown to play a critical role for normal brain function as it affects synaptic homeostasis and promotes learning and memory. Additionally, it has been implicated in pathologies, and most notably neurodegeneration and Alzheimer's disease. Hu et al. present an integrated mechanistic model of postsynaptic calcium concentration dynamics that incorporates various already published elementary models and demonstrate its accuracy using various experimental results. Notably, an important factor that impedes our understanding of the nervous system is its multiscale complexity. Building biologically accurate computational models that aim to span several scales quickly becomes impractical due to high computational load. This limitation holds true for the instantiation of many mechanistic models of calcium dynamics in simulations comprising a large number of neurons (each neuron containing thousands of synapses). To circumvent this limitation, Hu et al. also present an inputoutput model that accurately summarizes the functional dynamics of the mechanistic model, while significantly reducing the computational complexity, thereby enabling realistic large-scale simulations.

Visualizing and quantifying molecular interactions taking place at the synaptic level is a challenging task given the nanoscopic scale of the system, as well as the rapid temporal changes. Yet a growing number of tools now allow live recordings of various signaling pathways and proteinprotein interaction dynamics in time and space by ratiometric measurements, such as Bioluminescence Resonance Energy Transfer (BRET) Imaging. Chastagnier et al. present a singlecell BRET imaging protocol that is used to visualize proteinprotein interactions in living cells at subcellular level. The visualization technique, introduced by the same group in 2016, uses Nanoluciferase as a BRET donor partner in performing the imaging assay. Here, the authors present the steps needed to analyze BRET images concurrently with a toolset they have developed called "BRET-Analyzer" that facilitates systematic quantitative analysis.

The present research topic constitutes a collection of studies that investigate open questions on synaptic processes and their effect on information processing. We trust these will add to the collective knowledge and be of interest to scientists engaged in the topic.

### AUTHOR CONTRIBUTIONS

Both authors listed have made equal, direct and intellectual contribution to the work, and approved it for publication.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Di Maio and Bouteiller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Analog Signaling With the "Digital" Molecular Switch CaMKII

#### Stephen E. Clarke\*

Department of Bioengineering, Department of Neurosurgery, Stanford University, Stanford, CA, United States

Molecular switches, such as the protein kinase CaMKII, play a fundamental role in cell signaling by decoding inputs into either high or low states of activity; because the high activation state can be turned on and persist after the input ceases, these switches have earned a reputation as "digital." Although this on/off, binary perspective has been valuable for understanding long timescale synaptic plasticity, accumulating experimental evidence suggests that the CaMKII switch can also control plasticity on short timescales. To investigate this idea further, a non-autonomous, nonlinear ordinary differential equation, representative of a general bistable molecular switch, is analyzed. The results suggest that switch activity in regions surrounding either the high- or low-stable states of activation could act as a reliable analog signal, whose short timescale fluctuations relative to equilibrium track instantaneous input frequency. The model makes intriguing predictions and is validated against previous work demonstrating its suitability as a minimal representation of switch dynamics; in combination with existing experimental evidence, the theory suggests a multiplexed encoding of instantaneous frequency information over short timescales, with integration of total activity over longer timescales.

Keywords: molecular switches, frequency coding, stochastic resonance, cellular computation, CaMKII, synaptic plasticity, burst detection, hill function

# INTRODUCTION

Many cellular inputs lead to transient changes in cytosolic calcium (Ca2+) levels, generating temporally complex signals that reflect a wealth of information (Berridge et al., 2003). As such, cells express highly conserved molecular decoders capable of translating Ca2<sup>+</sup> oscillations into downstream signaling events that affect diverse processes such as gene transcription, development and aging, neural network homeostasis and the synaptic plasticity that underlies learning and memory (Lisman et al., 2002; Thomas and Huganir, 2004; Wen et al., 2004; Clapham, 2007; O'Leary et al., 2013; Tao et al., 2013; de Jong and Fioravante, 2014; Smedler and Uhlen, 2014). A celebrated example of a Ca2<sup>+</sup> decoder is the protein kinase Ca2+/calmodulin (CaM)-dependent protein kinase II (CaMKII; **Box 1**), which can be driven by transient levels of cytosolic Ca2<sup>+</sup> into either high or low states of switch-like activity. When stabilized through negative regulation by protein phosphatases, self-exciting (autophosphorylating) kinases such as CaMKII are an ideal component of signal

#### Edited by:

Vito Di Maio, Istituto di Scienze Applicate e Sistemi Intelligenti "Eduardo Caianiello," Italy

#### Reviewed by:

Joaquín J. Torres, Universidad de Granada, Spain Ingie Hong, Johns Hopkins Medicine, United States

#### \*Correspondence:

Stephen E. Clarke stclarke@stanford.edu

Received: 18 July 2018 Accepted: 31 October 2018 Published: 22 November 2018

#### Citation:

Clarke SE (2018) Analog Signaling With the "Digital" Molecular Switch CaMKII. Front. Comput. Neurosci. 12:92. doi: 10.3389/fncom.2018.00092

#### BOX 1 | The bistable molecular switch CaMKII and synaptic plasticity.

Accounting for approximately 1–2% of all brain protein, CaMKII is a central hub of cell signaling networks and can exert both pre- and post-synaptic control over information transmission in the central nervous system (Lisman et al., 2002). Once bound to the Ca2+-CaM complex, the kinase's ability to cooperatively autophosphorylate can produce two distinct stable states: either high or low levels of enzymatic activation. Postsynaptically, after repetitive stimulation, the high activation state may persist after the Ca2<sup>+</sup> signal subsides and can strengthen the connection between neurons, for example, the hippocampal CA3-CA1 synapses that support learning and memory (Lisman et al., 2012). However, it should be noted that the role of CaMKII autophosphorylation and constitutive activation is not fully understood or accepted (Michalski, 2013) and we are just beginning to gain better insight into the problem (Chang et al., 2017). This paper proposes that CaMKII's principal role is to meaningfully transmit information via its short term dynamics rather than store it permanently within levels of autonomously activated switch. Presynaptically, CaMKII also modifies connection strength (Wang and Maler, 1998; Ninan and Arancio, 2004; Pang et al., 2010). In weakly electric fish, the αCaMKII isoform produces presynaptic potentiation in a motion sensitive, excitatory sensory feedback pathway (Wang and Maler, 1998; Clarke and Maler, 2017). The kinase also potentiates hippocampal CA3-CA1 synapses, as evidenced by knocking-out αCaMKII, which leads to reduced synaptic potentiation under paired pulse facilitation protocols when compared to the wild-type (Chapman et al., 1995). Through enzymatic phosphorylation of voltage gated Ca2<sup>+</sup> channels and ryanodine receptors, αCaMKII can enhance Ca2<sup>+</sup> entry and Ca2+-induced Ca2<sup>+</sup> release in response to high frequency signals, potentially supporting hysteresis (Figure 1) and driving synaptic release (Catterall and Few, 2008). However, at the same CA3-CA1 synapses, post-tetanic potentiation protocols generate enhanced levels of potentiation in the same knock-out mice, illustrating that αCaMKII may also depress synaptic strength depending on the frequency and duration of the input (Chapman et al., 1995). Furthermore, αCaMKII has been shown to serve as a negative, activity-dependent regulator of neurotransmitter release probability at CA3-CA1 synapses (Hinds et al., 2003). This effect may be partially explained by the fact that CaMKII phosphorylates Ca <sup>2</sup>+-activated potassium channels that hyperpolarize the presynaptic terminal (Wang, 2008), decreasing the likelihood of Ca2<sup>+</sup> entry and evoked neurotransmitter release. Intriguingly, αCaMKII also plays a non-enzymatic role in presynaptic CA3-CA1 plasticity by regulating the number of docked synaptic vesicles containing neurotransmitter (Hojjati et al., 2007). In this case, decreased transmitter release could be explained by the fact that αCaMKII is acting as a sink for intracellular Ca2+, lowering the cytosolic levels that drive the machinery of synaptic vesicle fusion and influencing the size of the readily releasable vesicle pool (Thanawala and Regehr, 2013; Jackman et al., 2016). The size of the readily releasable pool is directly correlated with release probability at hippocampal synapses (Dobrunz and Stevens, 1997), supporting a putative role for αCaMKII in control of presynaptic plasticity parameters via Ca2<sup>+</sup> and CaM buffering (Hinds et al., 2003).

One of the most influential discoveries about CaMKII is its ability to decode the frequency of periodic Ca2<sup>+</sup> pulses into distinct amounts of long lasting, autonomously activated kinase (De Koninck and Schulman, 1998). However, the interpretation of CaMKII as a frequency decoder has been criticized based on the fact that mean values of activity, evoked by different combinations of Ca2<sup>+</sup> pulse size, duration and frequency, are ambiguously mapped into the same level of autonomously activated switch (Pinto et al., 2012), which suggests that the switch is actually integrating the Ca2<sup>+</sup> input over longer timescales. Alternatively, this article focuses on whether the concentration of activated switch acts as a reliable (analog) signal that reliably encodes frequency information over short timescales (sub-seconds), where Ca2<sup>+</sup> pulse size and duration are far more stable (Tank et al., 1995). The experimental evidence discussed above suggests that frequency coding by these "digital" molecular switches is more sophisticated than previously thought and that fast fluctuations in presynaptic αCaMKII around either the stable high- or low-activation state can better represent instantaneous frequency information, and, hypothetically, translate it into bidirectional control of synaptic strength in real-time.

amplification and have been previously likened to transistors on a computer chip, in that they may be turned on or off, presenting an ideal substrate for computation in cellular systems (Hunter, 1987; Ferrell and Ha, 2014).

The classic CaMKII experiments of De Koninck and Schulman provided the first demonstration that a molecular switch can decode the frequency of periodic Ca2<sup>+</sup> pulses into distinct, persistent levels of high enzymatic activation (De Koninck and Schulman, 1998). Although experimental evidence still largely lacks for whether this persistent activation occurs within functioning cells (Michalski, 2013), there are recent indications that it does occur to some extent (Michalski, 2014; Urakubo et al., 2014; Rossetti et al., 2017) and that autophosphorylation is key to this process (Chang et al., 2017; Rossetti et al., 2017). Many modeling studies of CaMKII autophosphorylation dynamics capture the ability of the high activation state to persist beyond the original Ca2<sup>+</sup> signal (known as hysteresis), which could potentially act over long timescales (seconds, minutes, and longer) (Zhabotinsky, 2000; Dupont et al., 2003; Graupner and Brunel, 2007). In these studies, the relationship between Ca2<sup>+</sup> concentration and the state of the molecular switch are determined from simulations of detailed, parameterized systems of differential equations that are not readily amenable to deeper mathematical analysis; furthermore, these studies are restricted to periodic inputs and concerned with long timescale activation. In order to better understand frequency coding over short timescales (milliseconds to seconds) and its putative effect on synaptic plasticity (**Box 1**), this article analyzes a reduced description of molecular switch behavior when subject to noisy, aperiodic forcing, while further demonstrating the model's compatibility with existing experimental and modeling results on CaMKII activation (De Koninck and Schulman, 1998; Dupont et al., 2003; Chang et al., 2017). As the study of cellular information processing shifts from individual transduction pathways, toward the emergent properties of complex signaling networks, simple mathematical models are becoming indispensable tools for both experimentalist and theoreticians alike by providing a trade-off between detailed performance and a reduced description that facilitates system-level studies (Bornholdt, 2005; Kotaleski and Blackwell, 2010). Much in the way that the leaky-integrate and fire model has benefited the study of spiking neurons (Jolivet et al., 2004; Burkitt, 2006), the minimal switch model discussed in this paper will hopefully facilitate further study of complex kinase-phosphatase networks.

### RESULTS

### A Bistable Switch Model

The following differential equation is an abstraction of a bistable molecular switch and was originally proposed as a model of genetic development by Lewis et al. (1977). This relatively

FIGURE 1 | Activation states of the bistable molecular switch model. (A) The model's potential function, U(x), visually describes the tendency for solutions to settle around one of two equilibrium points (x ∗ ), where the rate of change of switch activation, f(x), is 0 (parameters, r = 0.52 and c = 0.04). To the left of the stable equilibria (black circles), f(x) > 0 (green), and to the right, f(x) < 0 (blue), which forces perturbations to settle back into those states. Conversely, the sign of f(x) is reversed on both sides of the unstable equilibrium (red circle), such that tiny perturbations push the switch away, toward either stable state. (B) As r or c change, f(x) changes and can result in the loss of bistability. (i) To illustrate, r is fixed as the input c is varied: small values only support low activation, but, as c grows, bistability emerges and eventually disappears as only the high activation state is supported when c > cc (rightmost). A defining feature of bistability is the hysteresis effect, where the same value of a parameter may evoke different states depending on the history of activity. For example, the high activation state still exists for c less than the rightmost cc and can only be lost when c falls below the leftmost cc value. (ii) c is fixed while the negative regulation parameter r is varied. For small r, only the high activation state exists. As r grows larger, the system becomes bistable and, eventually, only the low state exists after crossing rc. Panel (iii) shows a parametric plot of the critical values cc(x) and rc(x), that partition the parameter space, and the bifurcation surface summarize the analysis completely (iv).

simple model is a useful analytical tool to understand the general properties of bistable kinetic systems and captures the qualitative dynamics of more complicated models of CaMKII (Zhabotinsky, 2000) (**Figure 1**). Although the model interpretation and results presented here are centered on CaMKII and synaptic plasticity, the reader is encouraged to consider the broader implications for instantaneous frequency coding with other molecular switches, such as mitogen-activated protein kinases (Xiong and Ferrell, 2003; Thomas and Huganir, 2004).

$$\frac{d\mathbf{y}}{dt} = k\_0 \mathbf{s} - k\_1 \mathbf{y} + \frac{k\_2 \mathbf{y}^n}{k\_3^n + \mathbf{y}^n}$$

In this formulation, the level of activated CaMKII (y) is stimulated by the presence of Ca2<sup>+</sup> bound to CaM, s, which will be studied as a function of time. For simplicity, it's assumed that pulses of Ca2<sup>+</sup> are bound upon cell entry, which is reasonable since CaM is found in large concentrations surrounding Ca2<sup>+</sup> channels and has a strong affinity for Ca2<sup>+</sup> (Chin and Means, 2000). Switch deactivation is directly proportional to the active CaMKII concentration at a rate k1, representing the activity of protein phosphatases. Finally, once activated, CaMKII has the ability to cooperatively bind Ca2+- CaM and autophosphorylate its own subunits, which motivates the nonlinear, positive feedback term captured by the Hill equation, where k<sup>2</sup> and k<sup>3</sup> are the association and dissociation constants, respectively. In addition to phosphorylation among the twelve subunits of a single CaMKII molecule, the ability to exchange active subunits between distinct CaMKII enzymes may connect this simple interpretation to a total, large pool of activated subunits distributed over multiple molecules (Stratton et al., 2014). Due to physiological constraints, y,s,k0,k1,k2,k<sup>3</sup> ≥ 0. In the following, this specific equation will be referred to as the full kinetic model.

The full kinetic model of Lewis et al. has been previously applied to bistable genetic networks (Lewis et al., 1977; Smolen et al., 1998; Zheng et al., 2011), transcriptional regulation (Heltberg et al., 2016; Kang et al., 2017), mitogen-activated protein kinases (Xiong and Ferrell, 2003), and incorporated into a larger phenomenological model of presynaptic plasticity (Oswald et al., 2002). Although insightful for their specific systems, these studies retain a large numbers of parameters that clutter analysis and obscure the generality of the results. Therefore, it is desirable to reduce the number of parameters and facilitate the following analysis by performing routine nondimensionalization. Let y = x · k3, r = k1k3 k2 , s = k2 k0 c and t = k3 k2 τ , which, when substituted into the original equation and simplifying gives the reduced but dynamically equivalent form:

$$\frac{d\mathbf{x}}{d\mathbf{r}} = \mathbf{c} - r\mathbf{x} + \frac{\mathbf{x}^n}{1 + \mathbf{x}^n} \tag{1}$$

This article is interested in a time varying c ≡ c<sup>0</sup> + c<sup>l</sup> (τ ), where c<sup>0</sup> reflects residual cytosolic Ca2+, whose slow dynamics are treated as fixed on the fast timescales over which the local Ca2<sup>+</sup> signal c<sup>l</sup> (τ ) fluctuates (Regehr, 2012). A timescale factor T = k3 k2 , the quotient of the switch deactivation and activation parameters, will be reintroduced later in order to connect the switch dynamics to time in seconds and stimulation frequency in Hz. The parameter r represents the kinetics of CaMKII subunit dephosphorylation by protein phosphatases and scales with the factor T. Finally, for highly cooperative reactions, n = 2 is a reasonable approximation of the Hill function exponent (Edelstein-Keshet, 2005) and a convention maintained by all of the studies listed above. The following bifurcation analysis is illustrated for n = 2, which allows for exact analytical solutions (**Figure 1** and Methods); however, the main results are then generalized to arbitrary <sup>n</sup> <sup>∈</sup> <sup>R</sup> +, which is much more realistic and has important consequences for frequency coding.

#### Stability and Bifurcation Analysis

Although interested in frequency-driven fluctuations over short timescales (**Box 1**), we must first examine the bistable, long timescale equilibrium behavior of the model that defines the switch's low and high activation states (Equation 1; **Figure 1**). An important reason for reducing the number of model parameters above is to simplify the analysis of all the possible system behaviors as a function of only a few parameter values. Having selected n = 2, we now only need to consider the effect of varying r and c; depending on their values, we may have one, two or three equilibrium points (x ∗ ), where the rate of change of the switch f(x) = c − rx + x 2 1+x 2 is equal to zero. For example, consider the values r = 0.52 and c = 0.04 that support bistability: there are three fixed points, two of which are stable, as illustrated by the switch's potential function U(x) = − R f(x)dx (**Figure 1A**). As r and c change, saddle node bifurcations can occur, resulting in the presence of only the high or low activation state. The corresponding bifurcation diagrams are displayed in **Figure 1B**; their derivation is found in the Methods section.

A key feature of bistability is the hysteresis effect, where the same value of a parameter may evoke different states depending on the history of activity. For example, as the Ca2<sup>+</sup> signal c increases, x ∗ grows larger until crossing the rightmost c<sup>c</sup> , where a saddle node bifurcation occurs and the switch jumps up to the high activation state, as the low state disappears (**Figure 1Bi**). Now, as c decreases back into the bistable range, the high activation state is preserved, and only lost when c crosses below the leftmost value of c<sup>c</sup> . This history dependent behavior is presumably central to sustained CaMKII activity on the order of seconds (Wang and Maler, 1998) (**Box 1**). A similar phenomenon occurs for the negative regulation parameter r (**Figure 1Bii**). The values of r<sup>c</sup> and c<sup>c</sup> are plotted parametrically as a function of the active switch in the bifurcation curves (**Figure 1Biii**). The bifurcation surface summarizes this information completely (**Figure 1Biv**).

#### Existence of Solutions Around Stable Equilibria

To date, studies of Lewis et al.'s full kinetic model have been restricted to static input and periodic forcing. It is of principal interest to characterize the model behavior in response to aperiodic forcing, in order to gain a more general, physiologically realistic understanding of frequency coding with molecular switches. In addition to potentially encoding frequency information into stable levels of activated switch for many seconds presynaptically (Wang and Maler, 1998), or minutes postsynaptically (Lisman et al., 2012), what about frequency coding on the order of milliseconds to seconds, which is associated with brief sequences of action potentialevoked Ca2<sup>+</sup> inputs? In a neighborhood surrounding a stable activation state (a sub-state region), is there a unique solution for a given time varying input signal? This question is not trivial, since small changes in the initial conditions of a nonlinear system (i.e., past switch activity) may generate drastically different behavior. Understanding the relationship that determines whether solutions converge or diverge around a given steady state could provide valuable insight into the properties of bistable molecular switches.

In the following section, we now reintroduce the scale factor T, since we are interested in studying frequency in Hz and time (t) in seconds. As such, Equation 1 becomes

$$T\frac{d\mathbf{x}}{dt} = c(t) - r\mathbf{x} + \frac{\mathbf{x}^n}{1 + \mathbf{x}^n} \tag{2}$$

First, to establish the existence of solutions around the high and low switch states, consider Equation 2 and note that f explicitly depends on the time-varying forcing term, c(t) ≡ c<sup>0</sup> + c<sup>l</sup> (t). The phosphatase activity r that can counteract the switch phosphorylation is treated as fixed. The function f(t, x(t)) is assumed to be Lipschitz continuous and well-defined within intervals of state space, y<sup>−</sup> ≤ x(t) ≤ y<sup>+</sup> satisfying the conditions <sup>f</sup>(t, <sup>y</sup>−) <sup>&</sup>gt; 0 and <sup>f</sup>(t, <sup>y</sup>+) <sup>&</sup>lt; 0 for all <sup>t</sup> <sup>∈</sup> <sup>R</sup> + (recall **Figure 1A**), which traps solutions within these boundaries. For any given point in time, there exist boundaries (y−, y+) determined by the parameters r, c0, and the input c<sup>l</sup> (t); we refer to values of the activated switch falling within these trapping regions as sub-state solutions, that is, fast timescale fluctuations that occur around either the high or low stable activation states (Lisman et al., 2012).

For (c,r) corresponding to the bistable region of parameter space (**Figure 1Biii**), there exist two intervals, x(t) ∈ (yl−, yl+) and x(t) ∈ (yh−, yh+), each surrounding one of the stable equilibrium points (x ∗ ). Now, we wish to locate values for the low state (yl−, yl+) and high state (yh−, yh+), where the existence of local time-varying solutions can be established. This problem is intimately linked to bifurcation, since yl<sup>+</sup> and yh−depend on the values of c and r. The choice of a lower bound for the interval that exists around the low activation state is yl−=0, since the physiological restriction c(t) ≥ 0 implies f(t, 0) > 0 for all <sup>t</sup> <sup>∈</sup> <sup>R</sup> <sup>+</sup>, ignoring the boring degenerate case of c(t) = x(t) = 0. The upper bound of the lower strip, yl+, can be chosen as a value x ∗ <sup>u</sup> − 1x, left of the unstable equilibrium x ∗ <sup>u</sup> where f(t, x ∗ <sup>u</sup>) = 0, such that 1c + f(t, x ∗ <sup>u</sup> − 1x) < 0; this condition ensures that the system is not trivially displaced into the high activation state by a single Ca2<sup>+</sup> pulse with amplitude 1c. For the high concentration strip (yh−, yh+), the lower bound yh<sup>−</sup> is chosen as a value of x infinitesimally greater than x ∗ <sup>u</sup>, that is, yh<sup>−</sup> = x ∗ <sup>u</sup> + ε for ε → 0. Since we have restricted r and c > c<sup>c</sup> (leftmost; **Figure 1B**) to the bistable range, we know that f(t, yh−) > 0 . For the upper bound of the high activation strip, it is enough to note that for x > x ∗ h , f(t, x(t)) < 0 and, since we wish to maximize the width of the strip, we take x arbitrarily large, denoting this value by yh<sup>+</sup> = x∞. During stimulation, if (c,r) drifts out of the bistable region of parameter space, a saddle node bifurcation occurs and only one interval exists; in this case, the bounds simply span the state space, y<sup>−</sup> = 0 and y<sup>+</sup> = x∞.

By invoking the Cauchy-Peano theorem, we guarantee the existence of at least one sub-state solution for every initial condition found within the interval regions defined above, since the conditions on the sign of the derivative f(t, x(t)) define trapping regions. However, this theorem says nothing about our interest in the fast timescales associated with short sequences of input pulses (100s of milliseconds), we assume that the Ca2<sup>+</sup> pulse size is fixed on this timescale, which is a reasonable approximation for hippocampal spiking frequencies less than 15 Hz (Tank et al., 1995). This distinction between short and long timescales provides a hypothetical means for the system to be less sensitive to variations in the Ca2<sup>+</sup> pulse size and the resulting frequency-intensity coding ambiguity [(Zhabotinsky, 2000; Pinto et al., 2012); see **Box 1**]. This could allow for more accurate representations of instantaneous frequency over short time periods, compared to long timescale frequency coding where input history, as well as additional adaptive and homeostatic processes may substantially adjust Ca2<sup>+</sup> signaling.

We now establish the stability and uniqueness of solutions for distinct initial conditions within a given interval of state space. Consider a general interval (y−, y+), where x(t) is a solution to Equation 2 with initial condition x<sup>0</sup> ∈ (y−, y+). Assume there is another solution, u(t), with a different initial condition u<sup>0</sup> ∈ (y−, y+). Writing z(t)=|u(t) − x(t)| and first assuming n is a positive integer, we see that

d dt <sup>z</sup>(t) <sup>=</sup> lim h→0 z(t + h) − z(t) h = lim h→0 <sup>u</sup>(<sup>t</sup> <sup>+</sup> <sup>h</sup>) <sup>−</sup> <sup>x</sup>(<sup>t</sup> <sup>+</sup> <sup>h</sup>)  − <sup>u</sup>(t) <sup>−</sup> <sup>x</sup>(t) h ≤ lim h→0 (u(<sup>t</sup> <sup>+</sup> <sup>h</sup>) <sup>−</sup> <sup>x</sup>(<sup>t</sup> <sup>+</sup> <sup>h</sup>)) <sup>−</sup> (u(t) <sup>−</sup> <sup>x</sup>(t)) h = lim h→0 (u(<sup>t</sup> <sup>+</sup> <sup>h</sup>) <sup>−</sup> <sup>u</sup>(t)) <sup>−</sup> (x(<sup>t</sup> <sup>+</sup> <sup>h</sup>) <sup>−</sup> <sup>x</sup>(t)) h = sgn[u(t) − x(t)] · d dt(u(t) <sup>−</sup> <sup>x</sup>(t)) = T −1 sgn[u(t) <sup>−</sup> <sup>x</sup>(t)] c(t) − r · u(t) + u n (t) 1 + u <sup>n</sup>(t) − c(t) − r · x(t) + x n (t) 1 + x <sup>n</sup>(t) = T −1 sgn[u(t) − x(t)] · −r · (u(t) − x(t)) + u n (t) − x n (t) (1 + u <sup>n</sup>(t))(1 <sup>+</sup> <sup>x</sup> <sup>n</sup>(t)) = T −1 sgn[u(t) − x(t)] · (u(t) − x(t)) · −r + u n (t) − x n (t) (u(t) − x(t))(1 + u <sup>n</sup>(t))(1 <sup>+</sup> <sup>x</sup> <sup>n</sup>(t)) = T −1 <sup>u</sup>(t) <sup>−</sup> <sup>x</sup>(t) · −<sup>r</sup> <sup>+</sup> Pn i=1 u i−1 (t) · x n−i (t) (1 + u <sup>n</sup>(t))(1 <sup>+</sup> <sup>x</sup> <sup>n</sup>(t)) forn ∈ Z + = T −1 z(t) · −r + p(u, x, n) 

whether solutions starting at different initial conditions will converge to a unique, stimulus-driven response that tracks changes in the Ca2<sup>+</sup> signal.

#### Uniqueness of Sub-State Solutions

As motivation for the following results, **Figure 2A** shows an example switch response to an 8 Hz Poisson pulse sequence, which is convolved with an alpha function filter (30 ms, Methods), then normalized to the signal's maximum and scaled by 1c = 0.5 to create an example input signal, which the switch tracks closely. Note, in this simulation, the alpha-function kernel was specifically chosen to be 30 ms based on literature values for the time course of local synaptic Ca2<sup>+</sup> signals (Sinha et al., 1997; Sabatini et al., 2002; Graupner and Brunel, 2012). Due to The expression p(u, x, n) achieves maximal values at intermediate switch levels that separate the low and high states of activation. Now, consider p(u, x, n) for the special case of n = 2 used in the bifurcation analysis; in this case, p(u, x, 2) = u+x (1+u 2 )(1+x 2 ) , which is plotted in **Figure 2B**. Setting the partial derivatives of the function to zero and solving for u and x, yields a critical point: (u, x) = <sup>√</sup> 3 3 , √ 3 3 . Substituting this into p gives a global maximum of <sup>3</sup> √ 3 8 . Since <sup>d</sup> dt z(t) ≤ T −1 −r + 3 √ 3 8 z(t) for all t, we can apply Grönwall's inequality, which gives us the following:

$$z(t) \le e^{T^{-1}} \int\_0^t (-r + \frac{3\sqrt{3}}{8}) ds$$

FIGURE 2 | Switch activity fluctuates with instantaneous input frequency. (A) Motivating example: switch response to an 8 Hz Poisson sequence of input pulses, convolved with an alpha function kernel to create a signal, c(t) = c<sup>0</sup> + c<sup>l</sup> (t). The switch's fluctuations track changes in the input frequency (n = 1.6, r = 0.61, c<sup>0</sup> = 0.04, and T = 0.01). (B) The example function p(u, x, 2) from the uniqueness proof achieves a maximum of 0.65; r must exceed this value to guarantee absolute convergence of the switch to a unique frequency-driven solution. (C) Initial conditions: u(0) = 1.7 and x(0) = 0.1. The value of T affects time-to-convergence between solutions and frequency filtering. From empirical studies, T ≤ 0.01 (Coultrap and Bayer, 2012). (D) (i) For <sup>r</sup> <sup>=</sup> 0.54 <sup>&</sup>lt; <sup>r</sup>c, sufficiently high frequency Ca2<sup>+</sup> pulses (bursts) cause transitions from the low to high state (illustrated for n = 2). By adjusting c<sup>0</sup> to take advantage of hysteresis, the cell can control whether or not it is sensitive to these burst-induced up states. The first two pulses (<10 Hz), where c<sup>0</sup> = 0, do not result in hysteresis, whereas the high frequency 10 Hz inter-pulse interval (right panel black arrow) with c0 augmented to 0.04 does; note that neither static value can generate the upstate alone without sufficient input (e.g., 5 Hz, left panel black arrow). The switch response differs during the transition between low and high states, but once settled around a given state gives good agreement (gray shading; the two example curves are compared by choosing an offset of 0.92 that minimizes the Euclidean distance between them). (ii) Top Simulation for r = 0.59 and n = 2, where x(t) has c<sup>0</sup> = 0.02 and thus cannot support bistability, vs. u(t) with c<sup>0</sup> = 0.04, which, when driven by input, traps the solutions around the high activation state through hysteresis. Under these conditions, convergence cannot occur. Bottom The absolute value of the difference between the relative changes in u(t) and x(t) induced respectively by the common input frequencies (1, see Results for details) plotted as a function of r; the maximum discrepancy of 0.046 is found at r = 0.59 and represents a small fraction of the total activated switch (Continued)

Substituting the expression for z(t) and solving this integral exponent yields,

$$|u(t) - x(t)| \le e^{-T^{-1}(r - \frac{3\sqrt{3}}{8})t}$$

and, as t → ∞ , we have

$$0 \le \lim\_{t \to \infty} |\mu(t) - \varkappa(t)| \le \lim\_{t \to \infty} e^{-T^{-1}(r - \frac{3\sqrt{3}}{8})t}$$

For r > 3 √ 3 8 (≈ 0.65), we obtain

$$0 \le \lim\_{t \to \infty} |u(t) - x(t)| \le 0$$

By the squeeze theorem we conclude that |u(t) − x(t)| → 0 as t → ∞ . Therefore, a unique frequency-driven solution exists and is independent of the initial conditions within the bounded interval. The time taken to converge to the unique solution is inversely proportional to T = k3 k2 (**Figure 2C**). The parameter value T = 0.01 s was chosen here for our specific example switch, CaMKII, whose dissociation constant (k3) has been experimentally determined to be at least 100 fold smaller than the activation constant (k2) that governs the rate of autophosphorylation (Coultrap and Bayer, 2012). Unlike the larger value of T = 0.1 s, T = 0.01 permits quick convergence and reliable encoding for the action potential frequencies characteristic of hippocampal CA3-CA1 synaptic input (approximately 1–15 Hz) (Mizuseki et al., 2012). Smaller values of T permit rapid convergence and more accurate frequency coding, but may become overly sensitive to temporary lulls in activity when c briefly drops below the leftmost critical value c<sup>c</sup> (recall **Figure 1Bi**).

It should be noted that r > 0.65 is an absolute guarantee of convergence to a unique frequency driven solution; but, from the bifurcation analysis (**Figure 1Biii**; Methods), we know that bistability does not exist for this value of r. However, in general, only −r + p(u, x, 2) < 0 is required, which, for low and high concentrations of activated switch, is obtained at smaller values of r that do support bistability. In fact, p(u, x, n) only exceeds the r value briefly during state transitions as it moves through the unstable equilibrium. Although a unique encoding of sub-state solutions can still exist for smaller r values around either the high or low state, convergence about the low activation state is now vulnerable to perturbation by short Ca2<sup>+</sup> inter-pulse intervals, thus acting as a high frequency event (burst) detector through induction of high switch activation. For example, experiments show that high frequency hippocampal activity (>15 Hz) causes successive Ca2<sup>+</sup> pulses to accumulate (Tank et al., 1995), which could effectively boost c<sup>0</sup> and serve to promote burst detection by transiently maintaining the high activation state via hysteresis (**Figure 2Di**). In theory, this dynamic burst threshold (the separatrix) is sensitive to recent levels of activation, and could be purposefully modulated by the cell through dynamic regulation of the parameters r and c<sup>0</sup> (Li et al., 2012). To restore the low state, the cell simply needs to adjust c<sup>0</sup> to fall below the leftmost critical value c<sup>c</sup> . The bottom panel of **Figure 2Di** illustrates that fluctuations around the high- and the low-stable states still yield nice agreement in their response to a given input frequency. Of course, during the state transition itself, the switch response can differ largely but once solutions are settled around their respective stable states the model appears to give good agreement (gray shading; the two example curves are compared by choosing an offset of 0.92 that minimizes the Euclidean distance between them).

When bistability is supported, the model response cannot always converge to an absolute level of activated switch, as illustrated in the top panel of **Figure 2Dii**; however, the fluctuations about the distinct stable states appear to be similar, as in **Figure 2Di**. To examine this idea further, repeated simulations of the model were performed, where x(t) has an associated residual Ca2<sup>+</sup> level of <sup>c</sup><sup>0</sup> <sup>=</sup> 0.02 and thus does not support bistability, vs. u(t) with c<sup>0</sup> = 0.04, which can trap the solution around the high activation state through hysteresis (**Figure 2Dii**, top). As was the case in **Figure 2C**, the same random spike sequences are used for x(t) and u(t) on each trial. For each inter-pulse interval of the repeated simulations, the change in the level of activated switch was computed as the difference between the switch activity sampled at the time of an input pulse and the subsequent maximum switch response that occurred before the next pulse. For each successive, shared inter-pulse interval, these differences, 1x and 1u, were determined separately for x(t) and u(t), then subtracted from each other for each 100 s trial, containing an average of 797 pulse intervals (8 Hz Poisson process). This was repeated 10 times for each parameter set and the composite mean of the absolute value of the difference between the change in the two solutions, 1 = |1u − 1x|, was determined as a function of r (**Figure 2Dii**, bottom). The maximum discrepancy between 1x and 1u, 0.046, occurs at r = 0.59 (used for **Figure 2Dii**, top) and is at least an order of magnitude less than typical values achieved in the low activation state. These results suggest that the relative change in switch activation about a stable state is generally quite consistent.

Realistically, the Hill function exponent n need not be restricted to integer values, which is unlikely in real biological systems. Thus, in the above proof, the expression p(u, x, n) is now left as <sup>u</sup> <sup>n</sup>−<sup>x</sup> n (u−x)(1+u <sup>n</sup>)(1+<sup>x</sup> n) for <sup>n</sup> <sup>∈</sup> <sup>R</sup> +, since there is no longer a closed form expression for the factorization of the numerator by the term u − x. The function p(u, x, n) has critical points at u = x, which occur at an apparent discontinuity due to the factor u − x in the denominator. However, assessing the limit as the difference between x and u becomes infinitesimally small, making the change of variable u = x + h as h → 0, and recognizing the limit definition of the power rule for differentiation, yields an expression for the maximum of <sup>p</sup>(u, <sup>x</sup>, <sup>n</sup>) for all <sup>u</sup>, <sup>x</sup>, <sup>n</sup> <sup>∈</sup> <sup>R</sup> +:

$$\begin{aligned} \max[p(u,x,n)] &= \lim\_{u \to \infty} p(u,x,n) \\ &= \lim\_{u \to \infty} \left[ \frac{u^n - \mathbf{x}^n}{(u-\mathbf{x})(1+u^n)(1+\mathbf{x}^n)} \right] \\ &= \lim\_{h \to 0} \left[ \frac{(\mathbf{x}+h)^n - \mathbf{x}^n}{(\mathbf{x}+h-\mathbf{x})} \right] \\ &\quad \cdot \lim\_{h \to 0} \frac{1}{(1+(\mathbf{x}+h)^n)(1+\mathbf{x}^n)} \\ &= \frac{d}{d\mathbf{x}} \left(\mathbf{x}^n\right) \cdot \frac{1}{(1+\mathbf{x}^n)^2} \\ &= \frac{n\mathbf{x}^{n-1}}{(1+\mathbf{x}^n)^2} \end{aligned}$$

For each value of the exponent n, the global maximum of this expression is determined for all <sup>x</sup> <sup>∈</sup> <sup>R</sup> +, and plotted (**Figure 2E**). Ignoring the highly uncooperative reaction exponents of n < 0.012, the global minimum of the class of functions p(u, x, n) is found at <sup>n</sup> <sup>=</sup> 1.55. Fascinatingly, the empirical <sup>α</sup>CaMKII data reported by De Koninck and Schulman was fit by a Hill function with an exponent of 1.6 (De Koninck and Schulman, 1998; Dupont et al., 2003). This intriguing match between their experiment and the model's theory suggests that αCaMKII's activation function may operate with this particular exponent as it provides the minimum level of negative regulation r required to maintain absolute convergence of unique input driven switch activity in the low activation state, even for intermediate levels of the switch response occurring just left of the unstable equilibrium (**Figure 1A**), where r must be much stronger to guarantee unique solutions (**Figure 2B**). As we will see in the following section, the value of n = 1.6 has additional benefits for amplifying the frequency response of weak calcium fluctuations in the presence of noise.

#### Molecular Switches and Stochastic Resonance

If Equation 2 is to capture actual molecular switch behavior in vivo, then we must understand frequency coding in the presence of biological noise. Given our interest in synaptic information transfer, it is natural to ask whether noise can improve the switch's frequency coding ability through stochastic resonance and how different combinations of our main parameters (for example the value of n) could potentially affect this phenomena. In particular, does the value n = 1.6 confer benefits for frequency coding? The results presented in this section are generated by Equation 2 with additive Ornstein-Uhlenbeck noise, η(t), which evolves according to the stochastic differential equation

$$\frac{d\eta}{dt} = -\frac{\eta}{\mathfrak{r}\_{\eta}} + \xi(t)$$

where ξ (t) is bounded Gaussian noise, N(0, 1), whose amplitude is scaled by a parameter σ. The simplest interpretation is that there is some weak noise in the Ca2<sup>+</sup> signal amplitude, which might arise from stochastic channel dynamics. The choice of the time constant τ<sup>n</sup> is based on previous studies of noisy microdomain Ca2<sup>+</sup> fluctuations, where an upper bound for the autocorrelation time was determined to be approximately 10 ms (von Wegner et al., 2014; Weinberg and Smith, 2014). This choice has the added benefit of matching our switch time constant T, should we instead assume the noise is inherent to switch activation, as well as matching a typical value for the membrane time constant of spiking neurons, whose noisy membrane potential fluctuations might influence the activity timescales of voltage-gated Ca2<sup>+</sup> channels.

**Figure 3A** shows the power spectrum (Pc) of a weak sinusoidal Ca2<sup>+</sup> oscillation, <sup>c</sup> <sup>=</sup> <sup>c</sup><sup>0</sup> <sup>+</sup> <sup>α</sup> sin(2πϕt), where <sup>c</sup>0<sup>=</sup> 0.04, <sup>α</sup> <sup>=</sup> 0.02 and ϕ = 2 Hz, which was selected based on the mean action potential frequency associated with the CA3 and CA1 regions of the hippocampus (Csicsvari et al., 2000). As expected, the noisy switch oscillates at the frequency ϕ , reflected in its power spectrum (Px). Very recently, the full kinetic model of Lewis et al., studied under the context of genetic regulation with n = 2, has been shown to produce the stochastic resonance effect (Kang et al., 2017), which is confirmed here for the dimensionally reduced model (Equation 2; **Figure 3B**). As σ increases from 0, frequency transfer, measured as the ratio of the switch power to signal power at ϕ , dips slightly and then improves dramatically, achieving a maximum at 0.29, followed by a quick decrease as the noise becomes dominant. When changing the exponent from n = 2 to n = 1.6, this spectral amplification becomes significantly larger, further suggesting that presynaptic α CaMKII functions as an important frequency decoder and that the exponent n = 1.6 may have evolved to fulfill this purpose. The reader should note that, for fair comparisons sake, r = 0.65 and r = 0.61 were selected respectively for n = 2 and n = 1.6 based on values obtained from **Figure 2E**, but this effect is qualitatively robust to changes in r and ϕ . Setting n = 1.6 also shifts the optimal noise strength to a substantially lower value, 0.09, which has the putative benefit of harnessing stochastic resonance and enhanced frequency representations for low intensity Ca2<sup>+</sup> signal noise.

The model results of Kang et al. (2017) depend on a full complement of parameters, which begs the question of whether stochastic resonance is a generic feature of the model switch or whether the effect is only significant for a certain range of the parameters. The dimensional reduction of the switch model performed here allows this question to be easily addressed as a function of the parameters c0, r and n. **Figure 3C** shows that the parameter r has significant influence over the value of σ that produces optimal spectral amplification and that, for some combinations of c0, r and n, the stochastic resonance effect disappears completely. The presence or absence of stochastic resonance may prove useful for deducing parameter ranges of molecular switches in vitro and in vivo. Furthermore, these noise fluctuations may generate unimodal (e.g., σ = 0.035) or bimodal (e.g., σ = 0.01) distributions of switch activation (**Figure 3D**, n = 1.6), which provides another experimentally testable prediction for αCaMKII, given that the switch state could control neurotransmitter release (see **Box 1**) and thus explain

FIGURE 3 | Frequency coding with noisy switches. (A) The switch model driven by a weak sinusoidal signal, <sup>c</sup>(t) <sup>=</sup> <sup>c</sup><sup>0</sup> <sup>+</sup> <sup>α</sup> sin(2πϕt), with <sup>c</sup><sup>0</sup> <sup>=</sup> 0.04, α = 0.02, ϕ = 2 Hz, and additive noise, η(t), whose intensity is scaled by the parameter σ and evolves according to τ<sup>η</sup> = 0.01. The switch amplifies the frequency content of the input, as shown by its power spectrum Px relative to the signal's, Pc. (B) Top: For n = 2, the ratio of switch power to signal power at ϕ is plotted as a function of the noise intensity σ , achieving a maximum at 0.29, that is, the switch displays stochastic resonance (SR). The value of σ that promotes optimal frequency transfer is denoted by σp. Bottom: For n = 1.6, there is substantially larger gain in the SR effect, and σp shifts to 0.09. (C) σp is plotted as a function of (n, r) and (c0, r), illustrating the presence or absence of SR. (D) For n = 1.6, stochastic switch simulations produce bimodal (e.g., σ = 0.01) or unimodal (e.g., σ = 0.035) activation around the low state (left column; r = 0.61, c<sup>0</sup> = 0.04) and the high state (right column; r = 0.54, c<sup>0</sup> = 0.04) (Box 1). Within each sub-state region, the input is uniquely encoded. (E) As model validation, the pulse duration (ms) and frequency experiments of De Koninck and Schulman were simulated (n = 1.6, r = 0.61, and T = 0.4), qualitatively capturing their results, as well as the results of a follow-up model (Dupont et al., 2003). The reader should note the ambiguity in autonomously activated (long timescale) switch activity, based on input duration and frequency (Pinto et al., 2012). (F) To validate the model against short timescale CaMKII data, a 1 Hz pulse train generates a calcium signal that drives the switch model: the top plot shows the entire period of stimulation and the decay of the switch after cessation of the stimulus (n = 1.6 and T = 0.1). These simulations qualitatively capture the exciting new experimental data of Chang et al. (2017). Note that, instead of a fixed value of negative regulation, r now linearly increases from 0.58 to 0.67 over the course of the stimulus, providing a potential explanation of the slow decay in the plateau switch concentration seen in the data but not in the model for a fixed value of r. The bottom plot zooms in on the first 15 s and also plots the model prediction for a T286A mutant form of CaMKII, which prevents the nonlinear switch activation. Naturally, this results in a much weaker response compared to the wild-type (WT), as seen in the data (Chang et al., 2017).

multimodal distributions of excitatory postsynaptic potential amplitudes (Larkman et al., 1992). The occupation of the low state (**Figure 3D**, left) vs. the high state (**Figure 3D**, right) depends on the level of negative regulation r and whether c<sup>0</sup> can support hysteresis: the parameter choices for the left column of **Figure 3D** do not support bistability (r = 0.61, c<sup>0</sup> = 0.04) and the switch fluctuates around the low activation state. The right column of **Figure 3D** does support bistability (r = 0.54,c<sup>0</sup> = 0.04) and input activity quickly drives high switch activation, while hysteresis ensures the switch stays within this state. Stochastic simulations for **Figure 3D** were performed by including additive Ornstein-Uhlenbeck noise, as described above. Further detail can be found in the Methods section.

#### Bridging Short Term Dynamics With Long Timescale Switch Activation

A potential caveat of the bistable switch model is that, even in the high activation state, the population of phosphorylated units (x) are still subject to the phosphatase activity (r). Equation 2 places difficult constraints on cells for long-timescale activation: if c<sup>0</sup> and r are not controlled carefully, the high activation state can be lost. Although high activation levels may only be short lived in vivo, it is important to establish a potential connection between the current model and existing theories of long timescale activation (**Box 1**; Introduction). Equation 2 effectively represents all of the phosphorylated subunits in a population of CaMKII molecules (each having twelve phosphorylation sites). When one of these dodecamers becomes fully phosphorylated, it could effectively become impervious to negative regulation by the phosphatases, since any cleaved subunit could immediately be re-phosphorylated by its neighboring subunits and the enzyme can be shielded by its interactions with downstream targets (e.g., an NMDA receptor subunit) (Lisman et al., 2012; Urakubo et al., 2014). Until now, the work presented here has ignored this potentially important feature of CaMKII, since the actual biological relevance of autonomous activation is still in question (**Box 1**). Therefore, to connect the short term dynamics to long timescales, we introduce a new variable (X) to represent the level of autonomously activated switch that might persist after the stimulus has been removed, even when Ca2<sup>+</sup> levels drop below the leftmost critical value c<sup>c</sup> that supports hysteresis (**Figure 1Bi**). X is calculated from Equation 2 by using Equation 3, explained below.

Motivated by the work of Pinto et al. (2012) (**Box 1**), let us assume that the total amount of autonomously activated switch (X) is simply proportional to the average amount of Ca2<sup>+</sup> input, which is determined by pulse amplitude, duration and frequency. As seen in **Figure 2**, this value is reflected by the amount of activated switch x(t) over the duration of the stimulus, 1t. Therefore, let X be the temporal average of x(t)

$$X = \left< \alpha \mathbf{x}(t) \right> = \frac{\alpha}{\Delta t} \int\_0^{\Delta t} \mathbf{x}(t) dt \tag{3}$$

The biological interpretation is as follows: at a given moment in time there is some likelihood for individual dodecamers to transition to the fully autonomous, phosphorylated switch state or bind to a downstream target. These autonomous elements accumulate over time. For simplicity, a fixed basal rate of transition of a given molecule to the fully autonomous state, ω, is assumed.

As a validation of the model's ability to produce CaMKIIlike behavior over long timescales, the essence of De Koninck and Schulman's experimental results (De Koninck and Schulman, 1998) and the model of (Dupont et al., 2003) are both captured qualitatively by Equations 2 and 3 (**Figure 3E**). Note that this result was generated using Equation 2 and 3, but does not include Ornstein-Uhlenbeck noise given the synthetic and controlled nature of the original experiment (De Koninck and Schulman, 1998). The timescale factor T was set on the order of 10−<sup>1</sup> s, which may reflect altered kinetics under the artificial conditions of the experiment, or the need for further refinement of the model presented here. For instance, the proportion ω is expected to grow larger as more of the subunit population becomes phosphorylated and cooperative activation grows stronger (Meyer et al., 1992; Chao et al., 2010), leading to an increased likelihood for individual dodecamers to transition to the fully autonomous state. This is expected to improve the reproduction of De Koninck and Schulman's results by flattening the curves at lower frequencies and steepening them at higher frequencies (De Koninck and Schulman, 1998). Future work should seek to determine ω(x), with the hopes of identifying reduced representations of strongly nonlinear CaMKII activation. In general, ω could also depend on the interaction of activated CaMKII with downstream targets; for example, in vitro evidence suggests constitutive CaMKII activation (hysteresis) requires interaction with an NMDA receptor peptide (Urakubo et al., 2014).

Given our interest in the short timescale behavior of the CaMKII switch, Equation 2 was further validated against recent experiments that used fluorescent life-time imaging microscopy to measure CaMKII activity with millisecond precision in neurons responding to glutamate uncaging (Chang et al., 2017). **Figure 3F** shows 1 Hz stimulation (30 pulses) that drives a dynamic calcium concentration, modeled as a first-order exponential decay with a time constant of 200 milliseconds. This signal (c; not shown) is used to drive the switch model with exponent n = 1.6. The value of T was set to 0.1 for this simulation; as an aside, T may be impacted by the mean frequencies experienced at the synapse over long timescales and can be modified according to the specifics of the system (e.g., prevs. post-synaptic), as well as further influenced by experimental conditions. These factors might explain the difference used to fit the Chang et al. (2017) from the value of T = 0.01 inferred from Coultrap and Bayer (2012). To account for the slow decay of plateaued switch activity (Chang et al., 2017), r was made to linearly increase for a small range over the course of stimulation. The justification for this is as follows: the protein phosphatase calcineurin has a much higher affinity for Ca2<sup>+</sup> ions than calmodulin, and will slowly strip Ca2<sup>+</sup> away from calmodulin that has yet to bind to CaMKII. Thus, it seems possible that the accumulation of activated calcineurin over the course of prolonged stimulation may exert a growing, adaptive effect on the switch. Although the linear change in r was chosen for simplicity, it's likely more complicated and may account for minor discrepancies between the model and the actual data of Chang et al. (2017). Finally, despite the fact that constitutively active CaMKII is unlikely to occur in vivo, the bistable-associated nonlinearity that governs its dynamics is still essential to its function. By repeating their experiment with a mutated form of CaMKII that cannot autophosphorylate (T286A), Chang et al. provide evidence that T286 phosphorylation is essential for the optimal integration of Ca2<sup>+</sup> signals by boosting the activation of the switch and slowing its decay. The bistable model nonlinearity in Equation 2 reflects this cooperative autophosphorylation – removing it results in a much weaker response (**Figure 3F**), as seen in the data (Chang et al., 2017). In addition to enabling the induction of long term plasticity postsynaptically, it is hypothesized that this nonlinearity is also essential for the switch to sequester cytosolic calcium and thus regulate neurotransmitter release at the presynaptic terminal.

#### DISCUSSION

A main goal of this study was to extend the frequency coding idea of De Koninck and Schulman (1998) in a generic switch model that captures the qualitative behavior of CaMKII, but focuses on fast timescale dynamics instead of slow timescales (**Box 1**). The model presented here may help to reconcile contradictory perspectives of CaMKII function (De Koninck and Schulman, 1998; Pinto et al., 2012) and suggests dual streams of information transfer that are temporally multiplexed: over short timescales, where the size and duration of the Ca2<sup>+</sup> pulse are more stable (Tank et al., 1995), the molecular switch can act as an encoder of instantaneous frequency information (e.g. **Figure 2A**) and function to bidirectionally regulate transmitter release at synapses through a combination of enzymatic and non-enzymatic activity (summarized in **Box 1**). Over longer timescales, the model switch integrates overall signal intensity, which could dictate long term changes in synaptic strength and is dependent on multiple factors such as slow Ca2+-induced Ca2<sup>+</sup> release (affecting c0) (Sharma and Vijayaraghavan, 2003; Catterall and Few, 2008), the size of the Ca2<sup>+</sup> pulse, its duration and the mean frequency of stimulation (**Figure 3E**).

Although the present work is a very preliminary investigation of the role of molecular switches in the processing of information in the brain, it provides some testable predictions for synaptic physiologists: establishing the presence of both bimodal and unimodal synaptic release that depends on αCaMKII and noise, as well as characterizing the hypothesized real-time modulation of release probability at central synapses by αCaMKII in response to natural, aperiodic stimulation patterns (specifically detection of bursting events). Finally, of particular interest, is the putative role of αCaMKII in the regulation of synchronous discharge probability and duration, as well as the propagation of CA3 oscillations into the CA1 area (Hinds et al., 2003). A more complete study including a coupling of the subcellular switch dynamics with those of the neuron and synapse will be published in the future.

Fascinatingly, the Hill function exponent of approximately 1.6 is not unique to CaMKII; calcium sensors within hippocampal basket cells display cooperative binding that also reflects the value 1.6, inferred from measurements of post-synaptic currents (Debanne et al., 2013). The mitogen-activated protein kinase (MEK1) is reported to have a Hill function exponent of approximately 1.7 (Ferrell and Ha, 2014), further hinting at the generality of the switch model. In general, bistable molecular switches such as CaMKII, are a conserved feature of cell signaling networks and generate combinatorial power in their collective action (Ferrell, 1997; Bhalla and Iyengar, 1999; Brandman et al., 2005). As previously described, stacking kinase pathways leads to an increase in the effective cooperative binding (described by the Hill function exponent) (Ferrell, 1997); for example, the extracellular signal-regulated kinase 2 (ERK2), which lies downstream of MEK1 is reported to have an approximately three-fold larger Hill coefficient of 4.9 (Ferrell and Ha, 2014), which is also associated with the famous Calyx of Held synapse (Debanne et al., 2013).

Due to their complex kinetics and network interactions, switch models are typically formulated by parameterized systems of differential equations that are not ideal for deeper mathematical analysis. It is proposed that the simple model described by Equation 2 can capture the core essence of molecular switches, much in the way that the leaky-integrate and fire model has been a successful abstraction of neuronal spiking activity, providing a trade-off between performance and a reduced description that facilitates network studies (Burkitt, 2006; Jolivet et al., 2008). This idea is supported by the inclusion of Equation 2 in an existing phenomenological model of feedback-driven synaptic plasticity, using the conventional exponent of n = 2 (Oswald et al., 2002). The relative simplicity of the switch model and its application to diverse signaling pathways make it a useful framework for further theoretical and experimental investigations into signaling networks, synaptic plasticity and cellular computation.

#### METHODS

#### Bifurcation Analysis

The first step of the bifurcation analysis is to find the equilibrium points. Setting n = 2, we rewrite Equation 1 as,

$$\frac{d\mathbf{x}}{dt} = \mathbf{g}(\mathbf{x}) - h(\mathbf{x})$$

where g(x) = x 2 1+x <sup>2</sup> and h(x) = rx − c. The fixed points occur when g(x) − h(x) = 0, which amounts to finding the solutions of the polynomial <sup>−</sup>rx<sup>3</sup> <sup>+</sup> (<sup>c</sup> <sup>+</sup> 1)<sup>x</sup> <sup>2</sup> <sup>−</sup> rx <sup>+</sup> <sup>c</sup> <sup>=</sup> 0. First, fix <sup>c</sup> and examine the effects of varying r. When c = 0, x = 0 is a fixed point, and, for a particular range of r, there exists two other positive valued fixed points, given by the roots of <sup>−</sup>rx2+x−<sup>r</sup> <sup>=</sup> 0. The critical value of the parameter r, denoted by r<sup>c</sup> is found by setting g(x) = h(x) and g ′ (x) = h ′ (x), which, when solved, gives

r<sup>c</sup> = x 1+x <sup>2</sup> = 2x (1+x 2 ) 2 . Three values of x satisfy this equality: −1, 0, and 1. Since we are not considering negative values of x, we have two critical points, r<sup>c</sup> = 0 and r<sup>c</sup> = 1 2 . Therefore, when c = 0, the system is bistable for 0 < r < 1 2 . For c > 0, r can be larger than <sup>1</sup> 2 while still preserving bistability (as in **Figure 1A**). We know r<sup>c</sup> occurs when h(x) = g(x) and h ′ (x) = g ′ (x); therefore, when h(x) > g(x) we lose a fixed point through a saddle node bifurcation. Forx > 0, the maximum of g(x) is found at x = q 1 3 which gives maxx[g(x)] = 3 √ 3 8 . Therefore, when r > r<sup>c</sup> = 3 √ 3 8 , only one fixed point exists.

Now, we are interested in fixing r and examining the effects of varying c. To find c<sup>c</sup> we set g(x) = h(x) and g ′ (x) = h ′ (x), which gives r = 2x (1+x 2 ) <sup>2</sup> and c<sup>c</sup> = rx − x 2 1+x 2 . Substituting the first expression into the second, we get c<sup>c</sup> = x 2 (1−x 2 ) (1+x 2 ) 2 . We differentiate with respect to x in order to locate the maximum value for c<sup>c</sup> ; 0 = 2x(1−3x 2 ) (1+x 2 ) 3 . This gives x = 0 and x = q 1 3 , which corresponds to c<sup>c</sup> = 0 and c<sup>c</sup> = 1 8 . When c > c<sup>c</sup> , only one fixed point exists for all values of r. For a fixed value of r that supports bistability, as c increases from 0 and crosses a critical value (cc), the fixed point x ∗ will jump up to the high amplitude branch. If c is now decreased, the fixed point remains on the high amplitude branch even as c becomes smaller than the corresponding c<sup>c</sup> . This hysteresis effect permits switch activation to remain as the transient Ca2<sup>+</sup> signal subsides, consistent with the findings from synaptic plasticity experiments (**Box 1**). Using the expressions derived for the critical values of r<sup>c</sup> and c<sup>c</sup> , we plot them parametrically as functions of x (**Figure 1Biii**). Saddle node bifurcations occur all along the boundary of these curves, it is here we find the values of r and c for which only two fixed points occur. Crossing each branch results in a pairwise collision and disappearance of two fixed points. Note where the bifurcation curves meet tangentially, (c,r) → 1 8 , 3 √ 3 8 , here we observe a co-dimension two bifurcation; beyond this point there is only one fixed point and the distinction between low and high activation states is blurred (**Figure 1Biii**).

#### Computational Specifications and Miscellaneous Details

Simulations were solved using the 4th order Runge-Kutta method, with the exception of the Ornstein-Uhlenbeck noise, which was solved using the stochastic Euler method (time step of 1 ms in all cases). All simulations were performed using custom

#### REFERENCES

Berridge, M. J., Bootman, M. D., and Roderick, H. L. (2003). Calcium signalling: dynamics, homeostasis and remodelling. Nat. Rev. Mol. Cell Biol. 4, 517–529. doi: 10.1038/nrm1155

Bornholdt, S. (2005). Less is more in modeling large genetic networks. Science 310, 449–451. doi: 10.1126/science.1119959

code, available upon request to the author, and were implemented on a Linux machine running Ubuntu 16.04 with an Intel core i7-6700 CPU, 3.4 GHz processing speed, and 62 GB of RAM.

Pulse train sequences {ti} were convolved with the filter t · e −(t−ti)/τ<sup>c</sup> , whose decay constant τ<sup>c</sup> was set to 30 ms, reflecting an accommodation of both pre- and post-synaptic calcium decay values from the literature that range from 15 to 43 ms (Sinha et al., 1997; Sabatini et al., 2002; Graupner and Brunel, 2012). The resulting input signal was normalized to the maximum value and then scaled by 1c. The decay value is closely related to the input frequencies typical of a given synapse and the definition of what constitutes a high frequency event in the system, since for events occurring faster than the decay, Ca2<sup>+</sup> accumulates quickly, driving the switch into the upstate. The putative burst detector will work for different τ<sup>c</sup> , but may require a different set of corresponding switch parameters, range of stimulation frequencies and pulse amplitudes.

Histogram bin sizes for **Figure 3D** were set using the Freedman-Diaconis method (Freedman and Diaconis, 1981).

#### AUTHOR'S NOTE

Bistable molecular switches can decode cellular inputs into distinct high- or low-states of persistent enzymatic activity. Although this on-off, "digital" perspective is valuable for long timescales, I suggest that short timescale fluctuations of switch activity around either stable state acts as an analog signal that reliably encodes instantaneous input frequency. A minimal model and theory make predictions about the molecular switch CaMKII, synaptic plasticity and burst detection.

#### AUTHOR CONTRIBUTIONS

The author confirms being the sole contributor of this work and has approved it for publication.

#### ACKNOWLEDGMENTS

I would like to thank Lin Wang for useful discussions and guidance during the early stages of the project, as well as Leonard Maler, Richard Naud, and Jean-Claude Béique for proofreading the manuscript and providing valuable comments. This work was funded by training grants awarded to SC from both the National Sciences and Engineering Research Council of Canada and the Canadian Institute of Health Research. A preprint of the manuscript is available on bioRxiv (Clarke, 2018).


Bhalla, U. S., and Iyengar, R. (1999). Emergent properties of networks of biological signaling pathways. Science 283, 381–387.


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Clarke. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The First 100 nm Inside the Pre-synaptic Terminal Where Calcium Diffusion Triggers Vesicular Release

Claire Guerrier <sup>1</sup> and David Holcman2,3 \*

<sup>1</sup> Department of Mathematics and Brain Research Center, University of British Columbia, Vancouver, BC, Canada, <sup>2</sup> Group of Applied Mathematics and Computational Biology, IBENS, École Normale Supérieure, Paris, France, <sup>3</sup> Churchill College, Cambridge University, Cambridge, United Kingdom

Calcium diffusion in the thin 100 nm layer located between the plasma membrane and docked vesicles in the pre-synaptic terminal of neuronal cells mediates vesicular fusion and synaptic transmission. Accounting for the narrow-cusp geometry located underneath the vesicle is a key ingredient that defines the probability and the time scale of calcium diffusion to bind calcium sensors for the initiation of vesicular release. We review here the time scale, the calcium binding dynamics and the consequences for asynchronous versus synchronous release. To conclude, three-dimensional modeling approaches and the associated coarse-grained simulations can now account efficiently for the precise co-organization of vesicles and Voltage-Gated-Calcium-Channel (VGCC). This co-organization is a key determinant of short-term plasticity and it shapes asynchronous release. Moreover, changing the location of VGCC from few nanometers underneath the vesicle modifies significantly the release probability. Finally, by modifying the calcium buffer concentration, a single synapse can switch from facilitation to depression.

#### Edited by:

Vito Di Maio, Istituto di Scienze Applicate e Sistemi Intelligenti "Eduardo Caianiello" (ISASI), Italy

#### Reviewed by:

Henrique Prado von Gersdorff, Oregon Health & Science University, United States Eric Hanse, University of Gothenburg, Sweden

> \*Correspondence: David Holcman holcman@biologie.ens.fr

Received: 01 February 2018 Accepted: 29 June 2018 Published: 23 July 2018

#### Citation:

Guerrier C and Holcman D (2018) The First 100 nm Inside the Pre-synaptic Terminal Where Calcium Diffusion Triggers Vesicular Release . Front. Synaptic Neurosci. 10:23. doi: 10.3389/fnsyn.2018.00023

Keywords: synaptic transmission, stochastic modeling, vesicular release, asynchronous release, facilitation, short-term plasticity (STP), residual calcium, simulations

### INTRODUCTION

The first 100 nm domain, between the plasma membrane and the vesicles in the pre-synaptic terminal remains difficult to study, yet it seems that a displacement as small as a 10 of nanometers in the molecular organization can affect vesicular release. There are many examples, where a 10 nm precision has to be achieved in order to guarantee normal physiology function. This is the case for the apposition of pre- and post-synaptic terminal of neuronal synapses: this apposition is obtained by a set of redundant adhesion molecules, such as laminins that self-organize to maintain the synapse structure and stability in the central nervous system. A lack of the laminin β<sup>2</sup> subunit leads to a disruption of the hippocampal synapse structure, to a misalignment of the pre- and post-synaptic partners and to an increased post-synaptic density (PSD) size (Egles et al., 2007). In addition, mutations in PSD proteins are associated with neurological and psychiatric diseases (Sheng and Kim, 2011). Another example is autism spectrum disorders which have been associated with the mutations in genes encoding Shank2 and Shank3, PSD-93, and a mis-regulation of adhesion molecules neuroligin 3, neuroligin 4, and neurexin 1 (Durand et al., 2011; Sheng and Kim, 2011), affecting a precise geometrical apposition.

In the pre-synaptic terminal, some vesicles are concentrated in a region called the Active Zone (AZ), which is well-aligned with the PSD of the post-synaptic terminal. This apposition creates geometrical columns: a single column alignment was originally hypothesized and numerical simulations showed that it maximizes the synaptic current (Xie et al., 1997) while minimizing its fluctuations (Taflia and Holcman, 2011). Multiple nanocolumns were predicted in Freche et al. (2011) to sustain synaptic response and a more reliable transmission compared to several synapses containing a single column. Finally, these columns have recently been confirmed experimentally and observed at super-resolution (Tang et al., 2016; Biederer et al., 2017). This nanocolumn example shows that synaptic transmission uses a tens of nanometer precision for its organization and for example a misalignment of synaptic terminals is at the basis of several pathological disorders (Südhof, 2008).

Another example is the PSD, that cannot permanently retain glutamatergic receptors that are moving by random motion (Nicoll, 2017). After a long enough time, these receptors spread out, modifying the synaptic current (Freche et al., 2011; Fresche et al., 2013). In the absence of direct experimental approaches, studying the functional consequences of the nanometer precision in the domain between the membrane and the vesicles at the presynaptic terminal has recently benefited from three-dimensional modeling and numerical simulations. We focus this review on this paradigm shift of analyzing the diffusion of calcium ions and in particular about the nano-metric relation between the organization of calcium channels and vesicles and how it shapes the release probability, synaptic transmission, asynchronous release, and short-term plasticity.

# 1. DYNAMICS AND CONSTANT RE-ORGANIZATION IN THE PRE-SYNAPTIC TERMINAL

Despite the fast advances of super-resolution microscopy, that allowed to reconstruct structural in vivo cell properties, or to follow calcium and voltage using genetically encoded indicators (Holcman and Yuste, 2015; Popovic et al., 2015; Beaulieu-Laroche and Harnett, 2018; Cartailler et al., 2018), it remains difficult to study the detailed molecular dynamics in nanometer domains at a time scale <100 ms. Indeed, to understand how molecules interact in nanometer domains, the notion of concentration has to be abandoned because it does not make much sense due to the large fluctuations in the small number of molecules. However, molecular interactions can still be transformed into a cellular activation at the micrometer level, but the exact biophysical mechanisms remain in most cases unclear or controversial. Modeling and numerical simulations based on biophysical principles have emerged as orthogonal tools compared to experiments to describe molecular dynamics at this spatio-temporal scales (Holcman and Schuss, 2015).

At this intermediate level between the molecular and the cellular scale, physical modeling of diffusion is based on Brownian motion, which requires to specify an inherent time scale of simulations. Indeed the motion of molecules follows a random walk approximation expressed by the Euler's scheme for a trajectory **X**(t) at time t:

$$\mathbf{X}(t+\Delta t) = \mathbf{X}(t) + \sqrt{2D(\mathbf{X})\Delta t} \text{ } \eta,\tag{1}$$

where η is a Gaussian variable, D(**X**) is the spatial dependent diffusion coefficient and 1t is the time scale to be chosen. It is usually a difficult choice. It should not be too small to avoid wasting simulation times and should not be too large compared to the small spatial scales involved in the microdomain, such as molecular binding sites.

In particular, taking into account in numerical simulations the region between vesicles and the plasma membrane has been particularly difficult to model because of its cusp-like geometry. It requires a specific mathematical treatment to estimate the mean time for a calcium ion after entering through a Voltage-Gated-Calcium-Channel (VGCC) to find a key calcium binding sensor, involved in triggering vesicular release (Guerrier and Holcman, 2014) (**Figure 1**). Such sites are Ca2+-binding proteins, located on synaptotagmins, that are involved in triggering directly or not vesicular fusion (Lee and Littleton, 2015). They are located precisely in this nanometric region below vesicles. Interestingly, spontaneous excitatory and inhibitory transmission are differently regulated by Ca2+ sensors (synapotagmin-1 and Doc2α/β a high-affinity Ca2+ sensors) (Courtney et al., 2018).

Calcium diffusion in the pre-synaptic terminal has traditionally been modeled as two- or three- dimensional diffusion (Modchang et al., 2010; Nadkarni et al., 2010; Stanley, 2016), but ignoring the three-dimensional complications of the vesicle shape. However, for auditory hair cells, Monte-Carlo simulations revealed (Graydon et al., 2011) that the spherical shape of the ribbon where vesicles are tethered, can generate a local Ca2<sup>+</sup> microdomain that enhances vesicular fusion by trapping calcium ions (Graydon et al., 2011, **Figure 3B**). This spherical ribbon that aggregates vesicles is likely to create an intermediate microdomain for calcium dynamics between the pre-synaptic bulk and the boundary layer near the membrane, which should be further investigated. For other types of synapses, while estimating the time scale of calcium binding, and computing the vesicular release probability, one cannot ignore the specific three-dimensional organization of the first 100 nm, the region underneath vesicles and the position of calcium sensors (Rothman et al., 2017). The release probability not only depends on the binding of calcium ions to sensor proteins, located underneath the vesicle (Guerrier and Holcman, 2015), but also on the AZ organization: a sparse vesicular distribution vs. vesicular crowding, and channels clustered vs. uniformly distributed (**Figure 1B**) (Delvendahl et al., 2015; Nakamura et al., 2015). The major components in these dependencies being the distance between VGCC and vesicles coupled to the particular cusp-like geometry (Guerrier and Holcman, 2015).

FIGURE 1 | Estimating the release probability. (A) Functional organization of the presynaptic terminal. An incoming action potential leads to the opening of voltage gated calcium channels (blue) located at the AZ (light blue). The consecutive entry of calcium ions (orange) triggers the fusion of docked vesicles (green) with the synaptic membrane, and the liberation of neurotransmitters (purple) in the synaptic cleft. The binding of neurotransmitters to specific receptors located in the post-synaptic terminal triggers the conversion of the chemical signal into an electrical signal in the post-synaptic neuron. (B) Model of the AZ organization. Vesicles (green) are regularly (left) or sparsely (right) distributed on a square lattice. Calcium channels (blue) can be clustered (left) or uniformly distributed (right) in the AZ. (C) Elementary 3-dimensional domain to compute the splitting probability for an ion starting in the bottom of the domain (blue circle representing a channel), to reach the target (red) before leaving the domain through the orange boundary. The other boundaries are reflecting. The vesicles are distributed on a square lattice of side 2H. (D) (Top): Probability to find three, four, or five calcium ions (full, dashed and dotted lines respectively) underneath a vesicle, in the case of sparse vesicular distribution: H = 100 nm (left) and in the case of crowding of vesicles at the AZ: H = 35 nm (right). The relation depends on the initial number of calcium ions. The diameter of the pre-synaptic vesicles is fixed at <sup>R</sup> <sup>=</sup> 40 nm (green), the diffusion coefficient for free calcium ions being <sup>D</sup>Ca <sup>=</sup> <sup>200</sup>µm<sup>2</sup> s −1 . (Bottom): Maximal channels distance r to activate the vesicle with a probability pact ≥ 0.8 (blue) and 0.2 (green), when there are N initial ions, for H = 100 nm (left) and H = 35 nm (right). We fix the threshold to three, four, or five calcium ions. The gray dashed line represents the maximal distance to the vesicle in the elementary domain: <sup>√</sup> 2H.

# 2. CALCIUM BINDING SENSORS AND VESICULAR RELEASE KINETICS

The synaptotagmin family of molecules are Ca2<sup>+</sup> sensors for vesicle fusion: following Ca2<sup>+</sup> binding, activation of the SNARE-complex mediates membrane fusion. Although there are two specific sensors on synaptotagmin, the number of ions necessary for fusion was estimated around 4 or 5 (Kochubey et al., 2011; Schneggenburger et al., 2012). Indeed, for example synaptotagmin1 has two Ca2+ binding C2 domains (C2A and C2B) and three Ca2+ bind in C2A and two Ca2+ bind in C2B (Gruget et al., 2018).

# 2.1. Modeling Calcium Binding and Limitation of Using a Forward Rate Constant

Modeling the causality between calcium dynamics and vesicular fusion relies on patching steps, resulting in simulations that do not necessarily account for the three-dimensional organization of the AZ. In Keller et al. (2015), by using the software Mcell, the forward rate constant of calcium ions is the reciprocal of the flux to the sensor targets is assumed rather than derived from physical considerations. This rate has to be pre-calculated especially when calcium sensors are located underneath vesicles (see next section). In addition, due to the vesicle crowding, the calcium flow cannot be represented by a Gaussian function, which is the classical probability density function of independent particles initially concentrated at one spot in a free space. This approximation further neglects the boundary effect and the depth of a synapse. In that context, providing numbers such as a distance of 30 nm for a possible exclusion area between vesicles and VGCCs, based on a two-dimensional approximation that ignores the effect of the vesicle size with a radius of R = 20 nm is not necessarily accurate. But the question is by how much? Clearly this geometrical limitation calls for a three-dimensional approach accounting for vesicular structure and organization.

A different modeling approach is described in Nakamura et al. (2015), based on diffusion (Nielsen et al., 2004; DiGregorio et al., 2007) of a two-dimensional coarse-grained lattice, using a two-dimensional Gaussian approximation for the initial calcium entrance. The vesicular release in these approaches is computed by using a Markov model, which is based on the concentration of calcium in a two-dimensional domain around the location of the channel. However, this concentration does fluctuate a lot, depending on the size of the sampling volume. This modeling approach neglects the small number of ions that penetrate underneath the vesicle, which is replaced by assuming a value for the forward rate. Indeed, to trigger vesicular release from the binding of calcium ions to sensors, a value should be given to the forward rate kon. Since the early work of von Smoluchowski in 1916 (Holcman and Schuss, 2017), this rate has been computed as the flux from a fixed concentration to a narrow window located on an infinite plan . But these assumptions are not satisfied for the small steady-state calcium concentration, because the rate should be computed for the first calcium arriving to the binding site, sampled from the transient entrance through the VGCC (Basnayake et al., 2017).

In summary, using a forward rate constant presuppose a geometrical organization. An alternative is to replace such a rate by Brownian simulations, but this approach is in general heavy computationally. Recent hybrid simulations have been developed, where classical diffusion is used far away from a sensor, while near the boundary of a channel a Brownian representation is used (see Dobramysl and Holcman, 2018 for a description of such framework).

# 2.2. How to Chose an Effective Diffusion Coefficient

What should be the buffer distribution and concentration in the pre-synaptic terminal? Various buffers and concentrations were previously considered: for the calyx of Held, a concentration of 400 µM immobile endogenous buffers plus a Parvalbumin-like Ca2<sup>+</sup> diffusing buffer with concentration of 50 µM were used in Keller et al. (2015), while ATP, a mobile Ca2<sup>+</sup> buffer, was present at a total concentration of 2 mM in all simulations (with a diffusion coefficient of 220 µm<sup>2</sup> /s). In Delvendahl et al. (2015) the concentration of fixed buffers is 480 µM, and of mobile buffers is 100 <sup>µ</sup><sup>M</sup> with <sup>D</sup> <sup>=</sup> <sup>20</sup> <sup>µ</sup>m<sup>2</sup> /s. The buffer capacity κ<sup>S</sup> (ratio of bound vs. free) is often chosen equal to 40. However, one of the main free parameter remaining is the value of the forward rate, which is valid for a single compartment model like the presynaptic terminal, which disregard its heterogeneity, but not for buffers located underneath vesicles.

### 2.3. Calcium-Buffer Interactions in the Pre-synaptic Bulk

When the pre-synaptic terminal is modeled as a bulk only, it does not matter that buffers are moving or not, because in one compartment the differential equations disregard the geometry. So, what matters is the number of free buffers available. If buffers are modeled with stochastic simulations, then space matters, especially during multiple entry of calcium channels, due to fluctuations of calcium buffers in the region very close to the calcium sensor sites underneath the vesicle. But to be efficient, this geometry should be implemented, which is often difficult.

In summary, buffers could be homogeneously distributed in the bulk, but between vesicles, the concentration is much less homogenous. More drastically, the number of buffers between the membrane and the first layer of vesicles can be of the order of a few: indeed, for vesicles positioned on a square lattice with radius 60nm with a height of 40nm, the volume of the parallelepiped lattice Ppara is V = 0.06 × 0.06 × 0.04 = 24 × <sup>6</sup> <sup>×</sup> <sup>10</sup>−<sup>6</sup> , minus the volume of a vesicle which is 32 <sup>×</sup> <sup>10</sup>−6µm<sup>3</sup> , that is Voltotal <sup>=</sup> 1.12 <sup>×</sup> <sup>10</sup>−4µm<sup>3</sup> . Inside such a region, for a buffer concentration of 40 µM, this represents around 26 buffers. For 400 µM (at the calyx of Held), this represents around 260 buffers. These numbers should be compared to the number of free calcium entry (from 80 to 500). It is conceivable that the different vesicular proteins located near the vesicular calcium sensor (others than the synaptotagmins) play a more important role for buffering calcium than the diffusing calcium buffers located in the bulk which can occasionally enter into the region Ppara, because they are precisely located at the right place and thus could create an efficient local calcium reservoir.

Finally, calcium mitochondria uptake can affect synaptic release through the MCU channels (Kwon et al., 2016). Mitochondrias participate in the calcium regulation that controls synaptic release and a MCU disruption could increase asynchronous release, decreasing the efficacy of synchronous neurotransmitter release and could also alter short-term presynaptic plasticity. This suggests that the distribution of mitochondria within the AZ could be as determinant as calcium buffers, a question that should be further investigated.

Another reason to reconsider the role of calcium buffer in the first 100 nm layer is the presence of an electric field that could push ions inside the bulk, as revealed recently for the synaptic terminal (Cartailler and Holcman, 2018; Cartailler et al., 2018). Too many buffers should disrupt calcium signaling and direct vesicular release. A low buffer capacity will increase asynchronous release. Thus, the concentration of buffers that favor a synchronous release should have an optimal value: not too low and not too large.

In Guerrier and Holcman (2015, 2016), the number of calcium entering through the VGCC vary from 80 to 500, which is also the case in Delvendahl et al. (2015), where they open around 12 Ca2<sup>+</sup> channels with a single channel current of 0.15 pA and a duration of 105 µs, leading to ≈ 500 − 600 calcium ions. Some of the 260 buffers with at least two binding sites, could bind calcium ions on their direct way to the calcium sensor underneath the vesicle. With ten times less buffers, much more free calcium would be available and then what would matter is the distance of the channel to the vesicle.

# 2.4. Phenomenological Laws Between Probability and the Overall Calcium Concentration

The relation between the molecular organization of VGCC, their numbers, calcium buffer dynamics, the release probability Prelease and the calcium flux of entering concentration of calcium [Ca]flux

mediated by an action potential remains an interesting problem. Over the years the following empirical relation has been proposed (Kochubey et al., 2011):

$$P\_{release} = k \left( [\text{Ca}]\_{flux} \right)^a,\tag{2}$$

where k is a constant. The relation between the exponent a and the number of active sensor binding sites is not direct, due to the effect of buffers, the clustering of VGCCs, but also the local vesicular geometry. Indeed, the contribution of geometry appears while computing the probability for an ion to go underneath the vesicle vs. going directly to the bulk (Guerrier and Holcman, 2014; Keller et al., 2015).

#### 2.5. Partial Conclusion: Modeling Vesicular Release

To conclude, there is not yet a derived formula from physical principle to connect the flux or transient calcium concentration and the release probability, however, stochastic approaches are used to estimate the arrival of ions to the calcium sensors (Delvendahl et al., 2015; Guerrier and Holcman, 2015, 2016; Nakamura et al., 2015). In addition, the three dimensional vesicular organization should be accounted either directly, by implementing vesicles as obstacles or by computing the Brownian flux to small targets located underneath.

Following calcium diffusion, once calcium ions are bound to buffers, they can possibly unbind, but often the exact value of the backward rate constant is unknown. In recent mathematical models (Guerrier and Holcman, 2015, 2016; Keller et al., 2015), vesicles are released when all 5 binding sites at a single sensor are occupied. If less than 5 calcium ions are bound, the vesicle is waiting for the final ions to arrive. There can be several copies of molecular sensors, but a single one might be sufficient to trigger release. It might also be conceivable that multiple binding sensors cooperate in the release process, and this possibility could explain the large modulation of the vesicular release probability (Kochubey et al., 2011; Schneggenburger et al., 2012).

We already emphasized that the calcium ions bound to calcium sensors located underneath the vesicles can contribute critically to the residual calcium ions pool, especially when the backward rate is very small. To conclude, there are two types of calcium ions contributing to shaping the vesicular release probability. The ions already bound to the specific calcium sensors located underneath the vesicles, and the ions freely moving in the pre-synaptic terminal, that can reach the calcium sensors or induce calcium release from organelles, hence filling a binding site, and ultimately triggering vesicular fusion. This mechanism represents a possible scenario for the calcium contribution to the asynchronous vesicular release (see below).

#### 3. DISTRIBUTION OF CALCIUM IONS ENTERING THROUGH A VGCC

The Hodgkin-Huxley model (Hodgkin and Huxley, 1952) can be used Guerrier and Holcman (2016) to generate a calcium influx current inside the pre-synaptic terminal. Following the opening of VGCC, this current corresponds to an entry per channel during a mean time of ≈1 ms for approximately 80 calcium ions, compatible with a previous estimation of 200 reviewed in Stanley (2016) or with the 45 ions per channels described in Keller et al. (2015): with a total of 12 channels, this would represent 540 ions entering during ≈ 0.1 ms. After entry, the calcium flux can be split into a ionic component that reaches the small region below the vesicle, and another one reaching the pre-synaptic bulk. The probability for a calcium ion to reach the binding region, defined as a small ribbon joining the vesicle and the plasma membrane, before the bulk, has been computed in models where vesicles are organized in a square lattice (**Figure 1B**) with length 2H, where H is of the order of the diameter of a vesicle, from 40 to 100 nm (**Figure 1C**). For a dense set of vesicles distributed on a square lattice, the splitting probability for a calcium ion (modeled as Brownian) to reach the ribbon before the bulk is (Guerrier and Holcman, 2015):

$$p\_{\mathcal{E}}(\mathbf{x}) = 1 - \frac{1 - A \frac{r\_{\nu\mathcal{E}}^2}{H^3}}{1 - \frac{2r\_{\nu\mathcal{E}}}{H^2}} \left(1 - \frac{2r\_{\nu\mathcal{E}}}{r(\mathbf{x})^2}\right),\tag{3}$$

where r<sup>v</sup> is the size of a vesicle, H is half the distance between two vesicles, A = 9.8, r(x) is the distance between the point source and the closest vesicle, and ε is the height of the small cylindrical ribbon (**Figure 1C**), where calcium sensors are located. This probability accounts for the particular geometry of the target and depends on the relative distance between the targets and the source points (Guerrier and Holcman, 2015).

## 3.1. Calcium Time Scales to the Ribbon Region

How long does it take for a calcium ion in the synaptic bulk or at the mouth of a VGCC (located far away from a vesicle) to enter into the cylindrical ribbon (red region in **Figures 1A–C**) underneath a vesicle? This mean time computed analytically in Guerrier and Holcman (2015) and Holcman and Schuss (2015) is given by:

$$
\bar{\mathfrak{r}} = \frac{|\Omega|}{4\pi D\mathfrak{s}}.\tag{4}
$$

For a volume of a pre-synaptic microdomain <sup>|</sup>| = <sup>1</sup> <sup>µ</sup>m<sup>3</sup> , a diffusion coefficient <sup>D</sup> <sup>=</sup> <sup>20</sup> <sup>µ</sup>m<sup>2</sup> /s, and a size of the ribbon in the range ε = 0.001 − 0.01 µm, the mean time is τ¯ = 3.6s. We note that the diffusion coefficient of calcium ions in a free environment is usually <sup>D</sup> <sup>≈</sup> <sup>200</sup>µm<sup>2</sup> /s, but the motion of ions in pre-synaptic terminals or in dendrites is restricted by obstacles such as microtubules, actin and organelles. In Biess et al. (2011), the effects of crowding on the diffusion coefficient has been estimated using modeling, simulations and a cytoplasmic fluid in a patch pipette, leading to a modified effective diffusion coefficient <sup>D</sup> <sup>≈</sup> <sup>20</sup>µm<sup>2</sup> /s.

However, the size of a calcium ion of the order of 1nm should not affect the classical law of diffusion. So the size of a calcium ion is often neglected in most modeling and stochastic simulations. Certainly, the most interesting part of vesicular crowding microenvironments is the local molecular organization underneath the vesicle, formed by all vesicular molecules, such as SNARE, syntaxin, or synaptotagmin, that could result in a 10–20 nm environment filled with polymer filaments. This intermediate spatial scale could have several effects such as (1) sequestrating calcium ions and/or creating channels to any sensor sites, (2) preventing calcium channels to get too close to the calcium sensors, (3) positioning synaptotagmin close to VGCC.

To conclude, as crowding is the main obstacle to diffusion, when the diffusion time scale involves long distances the effective diffusion coefficient should be used, while for short-distances containing little obstacles, computation should be performed with the cytoplasmic diffusion coefficient. No evidences have shown that changes of the cytoplasmic volumes, occurring at a time scale of milliseconds can modify the nature of calcium diffusion.

## 3.2. Direct and Indirect Vesicular Release Activation

Using the values mentioned in the previous subsection, the mean time for calcium to transit from the bulk underneath a vesicle is τ¯ = 1 kS = || <sup>4</sup>πD<sup>ε</sup> = 3.6 s (for ε = 1 nm) to 360 ms (for 10 nm). This mean time is much longer than the initial calcium transient from the channels(< 5 µs), as we shall see now. The reciprocal of this time is the Poissonian rate k<sup>S</sup> representing the rate of arrival of a free calcium ion to a binding sensor. However, this time is very different from the time for an ion entering through a VGCC close to the vesicle, to reach the region underneath the vesicle directly, i.e. while staying in a boundary layer around it. Indeed due to the confinement by the vesicle and in the absence of large obstacles at a distance of 20 nm, the time for an ion to hit a target sensor is τdirect ≈ l 2 4D . Using the free diffusion coefficient for calcium <sup>D</sup> <sup>≈</sup> <sup>200</sup>µm<sup>2</sup> /s, this leads to a mean time of τdirect ≈ 0.02<sup>2</sup> <sup>800</sup> = 0.5 µs. Thus when the channel is located very close to the vesicle, the direct binding of a single calcium is not time limiting. However, when the distance increases, the mean time for 5 calcium ions to arrive from a channel can be much longer (Guerrier and Holcman, 2015) of the order of 5–500µs, due to large fluctuations in the arrival time, the accumulation time for 5 ions (Dao Duc and Holcman, 2010) and the fast transient in the calcium available at the entrance of the channel. This scenario provides a physical mechanism for the fastest transmission events reported in Sabatini and Regehr (1996) and Von Gersdorff and Borst (2002).

# 4. COMPUTING THE RELEASE PROBABILITY WHEN VGCC ARE LOCATED UNDERNEATH A VESICLE

The probability that a finite number T of calcium ions (T = 3, 4, and 5 ions) are bound at specific binding sites located between a vesicle and the synaptic membrane (section 3.1), when N calcium ions have entered through a cluster or a single VGCC located at a distance r from the center of the closest vesicle is defined by:

$$p\_{act}(r, N) = \mathcal{P}(T \text{ ions have reached the synneptogram} | N \text{ ions})$$
 
$$\text{distance } r). \tag{5}$$

When the calcium unbinding events are too slow to be taken into account, the probability of vesicular release pact(r, N) is thus the one to find at least T ions inside the cylindrical ribbon. The probability to find exactly k ions out of N follows the Binomial distribution B(N, ps(r)), and the steady-state probability is:

$$p\_{\text{act}}(r,N) = \sum\_{k \ge T} \binom{N}{k} p\_s(r)^k \left(1 - p\_s(r)\right)^{N-k} \tag{6}$$

$$= 1 - \sum\_{k=0}^{T-1} \binom{N}{k} p\_s(r)^k \left(1 - p\_s(r)\right)^{N-k},$$

where ps(r) was computed in Equation (3). The maximal VGCC distance rmax,pact(N) to activate a vesicle with a probability pact ≥ 0.8 is shown in **Figure 1D**. The probability pact critically depends on the channels-vesicles distance, which can vary from a few to hundreds of nanometers. This fast decrease of the probability with the distance explains the large variability in the release probability as VGCC position can vary over time (Schneider et al., 2015).

The organization of vesicles in the AZ also influences the release probability: when vesicles are sparsely distributed (**Figures 1B–D**, H = 100 nm) and 100 ions entered through VGCC, then a 80% release probability pact = 0.8 is reached when the distance between the vesicles and the channels is smaller than 24 nm, compared to the 20 nm radius of the vesicle. This result shows that the co-localization of VGCC with a vesicle is a key feature determining a high synchronous release probability. However, for a high vesicular crowding (**Figures 1B–D**, described by choosing the distance H = 35 nm) and when 100 ions are released instantaneously at a VGCC, then the probability pact is higher than 0.9, regardless of the initial position of the channel, suggesting that vesicles are certainly released and leading to a synchronous release.

To conclude, a high crowding of vesicles should be associated with a high-release probability sustaining a synchronous release, while a sparse vesicle density might be associated with asynchronous release. Channels can be organized in clusters or uniformly distributed and this is also a major determinant governing release probability (**Figures 1B–D**). Indeed, the effect of channels clustering can be modeled by simply increasing the number of entering calcium ions. When vesicles are sparsely distributed, the 24 nm distance required to obtain a release probability pact = 0.8 when 100 ions are entering through one channel, is increased to 61 nm for 500 ions. This effect results from the local geometry of the ribbon underneath the vesicle. When the number of ions is low, this maximal distance to guarantee pact = 0.8 does not vary much when the activation threshold T increases from 3 to 5; however, for 500 ions, this distance changes significantly over 15 nm.

The maximal distance rmax,pact(N) between channels and vesicles to obtain a given release probability pact depends on the number N of entering ions. For a fixed probability pact, we plotted rmax,pact(N) in **Figure 1D**. For a sparse distribution of vesicles, characterized by a bulk located a distance H = 100 nm from the membrane, a vesicle is activated with a probability pact = 0.8 (resp. pact = 0.2), when 1,200 ions are entering at a distance 100 nm (resp. 450 ions), and 340 ions at a distance 50 nm (resp. 125 ions). This result has to be compared to the 200–500 nm diameter of the AZ (Südhof, 2012). Consequently, a sparse distribution of vesicles at the AZ requires a high number of entering calcium ions in order to trigger fusion, which can be achieved when channels are clustered. However, when channels are co-localized with vesicles, the activation probability pact is significantly increased: indeed 450 ions are necessary for activation for pact = 0.2 at a distance 100 nm. When the probability increases to 0.8, the distance reduces to 58 nm.

To conclude, a synapse with high release probability requires a nanometer precision of the channel location. However, this high requirement can be compensated by increasing the number of initial ions entering through VGCC clustering: with 2,000 ions, the maximum distance is relaxed to 140 nm. On the contrary, in a pre-synaptic terminal crowded at its surface with vesicles (characterized by H = 35 nm), very few initial ions are needed for an efficient release. Indeed, 50 ions are enough to activate a vesicle with probability 0.8, wherever the channels are located in AZ (**Figure 1D**).

#### 5. COMPUTING THE DISTRIBUTION OF RELEASE PROBABILITY

To compute the time distribution of the release probability and to account for the calcium ions at the AZ and in the bulk, a full model of the pre-synaptic terminal is needed. The main challenge for such derivation is to account both for the stochastic regime governed by rare events of individual calcium ions arriving to a sensor binding site, and the continuous description of the calcium concentration in the bulk of the synapse (Guerrier and Holcman, 2017). The classical approach consists in using partial differential equations that often cannot take into account easily the specific AZ organization and in particular the geometry near vesicles. An important assumption of these approaches is the use of the forward rate for the calcium to sensors. This rate is often assumed and not derived (contrary to the approach described in section 3.1).

To compute the sensor activation, a different approach is to use Monte-Carlo or Brownian simulations to follow each ionic trajectory. But this approach is often computationally greedy to detect the rare events of calcium hitting a small target (Holcman and Schuss, 2015). Recently, a hybrid Markovmass action model has been developed (Guerrier and Holcman, 2016), that combines a Markov chain to represent the stochastic events occurring at the AZ, with a mass-action laws model that represents calcium dynamics in the large bulk. The Markov chain and the mass-action model are coupled by the calcium ions coming from the bulk and binding to the sensor. The arrival time of such ions is Poissonian, with the rate computed taking into account the geometry of the vesicle, as discussed in section 3.1. This model is used to compute the time distribution of vesicular release (Guerrier and Holcman, 2016) and it shows that vesicular release is triggered by the binding of calcium ions that can originate either from the bulk or from VGCC.

The distribution of release time is bimodal although it is triggered by a single fast action potential (**Figure 2**). This simulation is initiated by three channels and each of them let a flow of 80 ions inside the cell during a time scale that was simulated from a Hudgkin-Huxley model (Guerrier and Holcman, 2016).

An example of specific simulation is as follows: the presynaptic terminal is a bulbous head of volume <sup>≈</sup> <sup>1</sup>µm<sup>3</sup> . At the AZ, we positioned eight vesicles, distributed on a square lattice of surface <sup>≈</sup> 0.13µm<sup>2</sup> (**Figure 1** and Guerrier and Holcman, 2016), so that the distance between two neighboring vesicles is 130nm, and each vesicle has a diameter of 40nm. The three calcium channels are uniformly distributed over the AZ, but remained from a distance of 6 − 10nm (we chose around 6nm here) from every vesicle. This distance corresponds to the radius of the red ribbon (**Figure 3A**) of height 1nm the calcium ions need to reach to simulate the binding to a sensor. For each simulation, the terminal undergoes three spikes at a fixed time interval 1t = 20−−150ms. Once a calcium ion enters the terminal, it can either reach a vesicle with probability p<sup>s</sup> (Equation 3), or enter inside the bulk with probability 1−p<sup>s</sup> according to the scheme of **Figure 3C**. We already discussed in subsection 2.2 how to chose the calcium diffusion coefficient: at the AZ it is 200µm<sup>2</sup> /s, and 20µm<sup>2</sup> /s in the bulk to account for crowding.

Inside the bulk, ions bind to buffers with a rate constant k<sup>0</sup> = 5.6s −1 and unbind with rate k−<sup>1</sup> = 500s −1 . Calcium can be extruded by pumps with a rate kpump = 0.88s <sup>−</sup><sup>1</sup> or they can leave the terminal with rate kes = 6.1s −1 . Finally, calcium can bind to the sensors located underneath a vesicle with a rate k<sup>S</sup> = 0.3s −1 . The number of buffer molecules in the bulk varies from 0 to 1,000. A calcium ion bound at the calcium sensor can unbind with a rate k<sup>U</sup> = 2000 s −1 (fast unbinding) or 5s −1 (slow unbinding). Once 5 calcium ions are bound to a calcium sensor, then the vesicle fuses with the synaptic membrane and the vesicle spot becomes free for a new vesicle coming from the recycling pool to bind with a rate <sup>1</sup> τDock (**Figure 3D**). An immediate refilling of vesicles at the AZ is obtained with a time τDock = 0 ms, but other delay are possible such as τDock = 50ms, or not refilling of vesicles with τDock = ∞ as shown in **Figure 4**.

In that simulation framework, the first peak in the time distribution of vesicular release follows a single stimulation and the second one in **Figure 2** (that is smaller in amplitude and wider) corresponds to the random arrival, over a much longer time period, of ions located in the synaptic terminal to small binding vesicular targets. To conclude, multiscale stochastic modeling approaches allow studying cellular events based on integrating discrete molecular events over long time scales from one milliseconds to seconds.

#### 5.1. Asynchronous Release

What defines the time lag between the arrival of an action potential and the first release of a vesicle? A possible mechanism is as follows: calcium ions flow inside a channel in less than 1ms. We saw above that calcium ions entering from VGCG located very close to a vesicle can reach key calcium sensors in less than

0.5µs. Either all calcium binding sites necessary to trigger fusion are now activated and fusion does occur, or some sites are still empty. In this case, the release will then depend on the arrival to the ribbon underneath the vesicle of calcium ions that will have to travel from other places, such as the bulk or other VGCC located far away. This second arrival process has a rate constant of few seconds (Guerrier and Holcman, 2016). These random arrival times of calcium ions to the vesicular calcium sensors define the distribution of vesicular release that can be widely spread due to the two distributions of calcium sources (**Figures 2A,B**). It also reveals that the distribution contains two peaks: one generated by immediate or synchronous release, which corresponds to the case where all calcium binding sites are immediately filled (Guerrier and Holcman, 2017), and the second release, which is asynchronous over hundreds of milliseconds, the time scale of which is defined by the arrival of far away calcium ions.

In that case the cusp geometry underneath the vesicle defines the arrival rate (see above and Guerrier and Holcman, 2015). In addition, it might be possible that vesicle-tethered and cytoplasmic Syntaxin1 proteins also contribute in differentially regulating synchronous versus asynchronous release kinetics (Lee and Littleton, 2015). In that case, the asynchronous release would be determined by the vesicle-tethered mechanism and not only by calcium arrival. The two processes could also combine together.

Under the calcium hypothesis controlling asynchronous release, increasing the concentration of calcium buffers in the bulk should reduce the amount of free calcium that can travel long distance in few milliseconds. Thus, increasing calcium buffer concentration should reduce asynchronous release, as shown in numerical simulations (**Figures 2A,B** Guerrier and Holcman, 2016), and experimentally (Fawley et al., 2016). Indeed, this hypothesis has received more support as buffering intracellular calcium with EGTA-AM reduced asynchronous EPSC. Asynchronous or spontaneous release involve calcium coming from the bulk or VGCC located far away from the vesicles (Goswami et al., 2012; Dai et al., 2015; Stanley, 2016). The release of asynchronous vesicles was largely diminished when calcium chelation such as BAPTA (Fawley et al., 2016) was used. The authors of that study concluded that asynchronous release should rely on calcium ions involving longer trajectories compared to the ones originating from VGCC located near a vesicle. Note that the number of buffer molecules such as calmodulin in the first 100 nm between the vesicle and the membrane is very small (of the order of a few) and thus these molecules do not affect synchronous release, as shown also experimentally in Fawley et al. (2016). However, it remains unclear whether or not the readily releasable pool organization can influence asynchronous release.

To conclude, VGCC located underneath vesicles are not the only contributor filling the calcium binding sites, required for vesicular fusion. Actually, not all VGCC are located underneath vesicles. If one or a cluster of VGCC are not close enough or are moved away from a vesicle of a distance of 10 or 20 nm, the probability to have the correct amount of calcium ions on the sensor binding sites can decrease significantly (**Figure 1D**) and thus vesicular release will have to involve calcium ions coming from far away. Modeling and experiments (Fawley et al., 2016) are now converging and suggest that this second source

of calcium defines and regulates asynchronous release, when calcium ions generated from local VGCC is not enough. In that context, any vesicle can potentially lead to an asynchronous release as long as it does not contain enough VGCC underneath. Most likely, these vesicles are located at the periphery of the AZ, where the density of VGCC could decrease. The exact relation between VGCC distribution and vesicular organization remains unclear.

#### 5.2. Simulating Multiple Spikes and Paired-Pulse Ratio

The model developed in Guerrier and Holcman (2016, 2017) can also be used to explore the short-term synaptic properties such as calcium accumulation and the distribution of time for vesicular release. First the method is consistent with any other simulation methods, second it is possible to test how the backward rate constant of the calcium ion to the sensor affects the time distribution of release, as well as the paired-pulse ratio (PPR), computed in this case as the ratio, after two consecutive spikes of the amount of fused vesicles after the second spike divided by the first one. A PPR >1 means that the release probability is increased which is usually interpreted as short-term synaptic facilitation, while a PPR <1 corresponds to a decrease, interpreted as short-term synaptic depression.

For fast calcium unbinding to the sensors, there is no accumulation of calcium in the sensor site and thus the release probability is independent of the spike train and of the buffer concentration (**Figure 4A**, thus PPR = 1). Conversely, a slow unbinding time from the sensors is associated with a local increase in the release probability (PPR >1) (**Figure 4B**). This facilitation is due to various sources of calcium: first the ones already bound to sensors and second to calcium accumulation in the bulk following multiple spikes. This second source is diminished by increasing the amount of calcium buffers, which can lead at high buffer concentration to a decrease in the release probability PPR < 1 (**Figure 4B**, bottom).

Hence, by changing the buffer concentration, a synapse can go from a facilitating state to a depressing state. To investigate the role of the readily-releasable-pool organization in the release probability, a first step is to use a vesicular replacement rate at the AZ by considering the time τDock for a vesicle from the readily releasable pool of vesicles to replace a vesicle that has just fused: in the extreme case τDock = 0, which corresponds to an immediate refilling of vesicles, and τDock = ∞ which corresponds to no refilling of vesicles: **Figure 4C** shows the behavior of a vesicular release with no refilling of vesicles, and a slow unbinding rate. Simulations show an increase in release probability after the second spikes, due to the calcium accumulation at the binding sites. After the third spike, the release probability is decreased, due to the lack of vesicles docked at the AZ, which would be interpreted as short-term depression. To conclude, a low unbinding rate is responsible for calcium accumulation in the sensor binding site that increases the release probability and defines short-term facilitation.

#### 6. CONCLUSION AND PERSPECTIVE

The lesson from modeling and numerical simulations of diffusion in the first hundred nanometers between docked vesicles and the plasma membrane is that this boundary layer is crucial for computing the vesicular release probability due to the critical position of vesicular sensors. This space is difficult to access experimentally and its role has been underestimated in shortterm plasticity. However, experimental approaches using calcium chelator such as BAPTA or EGTA confirm the role of calcium ions traveling from far away compared to the ones entering directly through VGCC located underneath a vesicle to trigger release.

Another key feature relevant for short-term plasticity is the structural organization and the spatial correlation between the distribution of VGCCs and vesicles. Do vesicles contain the same amount of close VGCCs? What defines the exact location of VGCC underneath vesicles? Can this number fluctuate? What happens after vesicular fusion? How are VGCC redistributed? It is possible that VGCCs are constantly moving to find local optimal sites location (Schneider et al., 2015).

Short-term facilitation is classically thought as the accumulation of calcium in the synaptic bulk (calcium hypothesis

of Katz and Miledi, 1968), due to various possibilities such as slow and fast buffers. But computational evidences (Guerrier and Holcman, 2017) reviewed here suggest that calcium accumulation at the sensor binding sites and not in the bulk is actually the determinant effect to pre-activate vesicular release (by binding a certain fraction of the sensor sites). A similar conclusion was reached in Bornschein et al. (2013) using fluctuation analysis, calcium imaging and numerical simulation analysis indicating that the residual calcium bound to the release sensors (see also Schneggenburger and Neher, 2000), after the first AP could cause Paired Pulse Facilitation at Purkinje neuron synapses.

A byproduct of facilitation should be asynchronous release, because the random calcium accumulation at sensor increases the time window when a vesicular can be released, due to ions arriving at random time from the bulk, thus leading to a high variability in the calcium arrival and the vesicular release times. However, synaptic facilitation requires a low concentration of calcium buffer, suggesting that for facilitating synapses, calcium buffers should be maintained at a low level. In general, the relation between facilitation and asynchronous release triggered by residual calcium can be mediated by the specific molecular composition of the vesicle: indeed synaptotagmin 7 (Syt7) (Turecek and Regehr, 2018) present at cerebellar granule cell synapses onto stellate cells and Purkinje cells (mice) plays such a role: in Syt7 KO, facilitation, and asynchronous release are smaller and shorter lived than in WT, although the residual calcium was unchanged. This is in contrast with synchronous released mediated by synaptotagmin I. It would be certainly interesting to model the distribution of various synaptotagmin on vesicles to determine when a vesicle will be more likely to be released asynchronously vs synchronously. In particular, is the distribution of these two SytI and 7 different on vesicles? Do they have very different calcium buffer affinity? These properties could be explored using numerical simulations.

In terms of calcium residual regulation, a low concentration level can be achieved by preventing ER or mitochondria to come in too close proximity of the AZ. In contrast, synchronous release is associated with a high calcium buffer concentration, preventing calcium ions to travel from far away. More specifically, twodimensional numerical simulations (Delvendahl et al., 2015), modeling essentially the synaptic bulk, revealed that fixed endogenous buffers with low affinity, characterized by a low calcium-binding ratio, mixed with mobile buffers with high affinity, results in a fast AZ calcium clearance. This results in synchronous high-frequency transmission (at 200 Hz). But it remains unclear how calcium fluctuations is maintained low (Modchang et al., 2010; Nadkarni et al., 2010; Dittrich et al., 2013; Weinberg, 2015) especially near the vesicular calcium sensor.

Finally, how the rate of vesicular release can vary over 6 orders of magnitude for the same synapse (Kochubey et al., 2011) also remains enigmatic. The cusp geometry and rare binding events may hold the key to the solution of this spectacular modulation of the vesicular release rate (Guerrier and Holcman, 2015).

#### REFERENCES


Modeling vesicular trafficking and recycling at various synapses including ribbon synapses (Thoreson et al., 2016) should clarify the organization of the pre-synaptic terminal (Graydon et al., 2011, 2014) and the effect of VGCC trafficking (Schneider et al., 2015).

## AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

## ACKNOWLEDGMENTS

We thank S. Hallermann for reviewing this manuscript and for pointing us his articles, that we now discuss here. CG would like to thank the Fyssen foundation for support.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Guerrier and Holcman. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# A Glutamatergic Spine Model to Enable Multi-Scale Modeling of Nonlinear Calcium Dynamics

Eric Hu\*, Adam Mergenthal, Clayton S. Bingham, Dong Song, Jean-Marie Bouteiller\* and Theodore W. Berger

Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, United States

In synapses, calcium is required for modulating synaptic transmission, plasticity, synaptogenesis, and synaptic pruning. The regulation of calcium dynamics within neurons involves cellular mechanisms such as synaptically activated channels and pumps, calcium buffers, and calcium sequestrating organelles. Many experimental studies tend to focus on only one or a small number of these mechanisms, as technical limitations make it difficult to observe all features at once. Computational modeling enables incorporation of many of these properties together, allowing for more complete and integrated studies. However, the scale of existing detailed models is often limited to synaptic and dendritic compartments as the computational burden rapidly increases when these models are integrated in cellular or network level simulations. In this article we present a computational model of calcium dynamics at the postsynaptic spine of a CA1 pyramidal neuron, as well as a methodology that enables its implementation in multi-scale, large-scale simulations. We first present a mechanistic model that includes individually validated models of various components involved in the regulation of calcium at the spine. We validated our mechanistic model by comparing simulated calcium levels to experimental data found in the literature. We performed additional simulations with the mechanistic model to determine how the simulated calcium activity varies with respect to presynaptic-postsynaptic stimulation intervals and spine distance from the soma. We then developed an input-output (IO) model that complements the mechanistic calcium model and provide a computationally efficient representation for use in larger scale modeling studies; we show the performance of the IO model compared to the mechanistic model in terms of accuracy and speed. The models presented here help achieve two objectives. First, the mechanistic model provides a comprehensive platform to describe spine calcium dynamics based on individual contributing factors. Second, the IO model is trained on the main dynamical features of the mechanistic model and enables nonlinear spine calcium modeling on the cell and network level simulation scales. Utilizing both model representations provide a multi-level perspective on calcium dynamics, originating from the molecular interactions at spines and propagating the effects to higher levels of activity involved in network behavior.

Keywords: multi-scale modeling, spine calcium, computational model, nonlinear dynamical systems, glutamatergic synapse

#### Edited by:

Florentin Wörgötter, Georg-August-Universität Göttingen, Germany

#### Reviewed by:

Christian Tetzlaff, Max-Planck-Institut für Dynamik und Selbstorganisation, Germany Michael Fauth, University of Edinburgh, United Kingdom

#### \*Correspondence:

Eric Hu ehu@usc.edu Jean-Marie Bouteiller jbouteil@usc.edu

Received: 28 December 2017 Accepted: 05 July 2018 Published: 27 July 2018

#### Citation:

Hu E, Mergenthal A, Bingham CS, Song D, Bouteiller J-M and Berger TW (2018) A Glutamatergic Spine Model to Enable Multi-Scale Modeling of Nonlinear Calcium Dynamics. Front. Comput. Neurosci. 12:58. doi: 10.3389/fncom.2018.00058

# INTRODUCTION

The calcium ion is a key biochemical signaling molecule for cellular function, and a number of studies have demonstrated its importance in numerous cell types. Calcium is known to be involved in regulation of gene transcription factors (Bading et al., 1993; Dolmetsch et al., 1998), muscle contraction (Ebashi and Endo, 1968; Weber and Murray, 1973), bone formation (Zhu and Prince, 2012), cell metabolism (Contreras et al., 2010) and apoptosis (Mattson and Chan, 2003). Within neurons, calcium has an especially critical role in modulating communication and network activity (Zucker, 1999; Emptage et al., 2001). Calcium has been extensively investigated in postsynaptic spines due to its involvement in various signaling cascades that lead to synapse formation and plasticity, cellular mechanisms which underlie learning and memory.

Early experimental studies on spine calcium focused on measuring calcium concentration changes at postsynaptic spines of the CA1 pyramidal cell in response to presynaptic stimulations and action potential transients through fluorescence procedures (Sabatini et al., 2002). Other studies focused on identifying sources of calcium influx, which include voltage dependent calcium channels (Bloodgood and Sabatini, 2007), intracellular calcium stores (Holbro et al., 2009), and NMDA receptor channels (Bloodgood and Sabatini, 2009). Downstream calcium signaling pathways have also been investigated, where calcium microdomains and localized calcium signaling (Higley and Sabatini, 2012) can invoke signaling of the ubiquitous Calmodulin/CAMKII protein, which leads to AMPA receptor upregulation (Naoki et al., 2005; Zhabotinsky et al., 2006). These standalone studies have helped further our understanding of numerous processes that regulate calcium dynamics in spines, but more research is needed to study how such processes interact with each other and together influence synaptic transmission.

Computational models have also been successfully adapted to the study of calcium dynamics in spines and neurons (Shouval et al., 2002; Naoki et al., 2005; Bartol et al., 2015). The advantage of using computer models over experimental protocols is their inherent ability to provide a controlled environment and overcome limitations in size constraints, an issue common when studying subcellular spaces such as synaptic compartments. Calcium dynamics models vary in biophysiological detail and accuracy, ranging from simple phenomonological models that directly relate calcium concentration to synaptic plasticity (Shouval et al., 2002), to detailed and complete reconstruction of the calcium dynamics at a dendritic subsection of a hippocampal CA1 neuron using stochastic Monte Carlo simulations (Bartol et al., 2015). Typically, the degree of realism used for a model is often dependent on the scale of the study, where calcium models at the cellular or network scales have less physiological detail than molecular scale models.

Scientific computation has been trending toward multiscale modeling, where models are developed to explore biological phenomena across multiple length or hierarchical scales (Yu and Bagheri, 2016; Seo and Jun, 2017). Neural computation spans molecular (Naoki et al., 2005; Bartol et al., 2015), cellular (Jarsky et al., 2005; Migliore and Migliore, 2012), network (Hendrickson et al., 2015), and cortical systems (Markram, 2006) hierarchical scales. Thus, multi-scale modeling is especially valuable for studying calcium dynamics in neurons, because calcium induced molecular signaling cascades can have dramatic effects on patterns of activity at the cellular/network level. Calcium activity in a CA1 pyramidal neuron is not distributed equally as evidenced in experimental studies (Higley and Sabatini, 2008). Rather, calcium processes can be categorized based on location and degree of influence (**Figure 1**). For example, the high-voltage activated (HVA) calcium channels present on postsynaptic spines can create localized calcium microdomains—brief, high concentrations of calcium in a small area—which can then lead to protein kinase activation and induce secondary messenger cascades, eventually resulting in synaptic plasticity and larger scale changes in cell properties and network activity (Higley and Sabatini, 2012). Meanwhile, intracellular calcium stores, such as the smooth endoplasmic reticulum (ER) and mitochondria, in dendritic compartments can be involved in more metabolic processes such as gene transcription and ATP production (Li et al., 2004). If possible, use of detailed, biologically accurate computational models in large-scale simulations would add the benefit of monitoring such significant molecular-level influences in large network interaction; however, simulating such a large number of complex mechanisms and interactions in a realistic model requires a prohibitively high computational cost. Unfortunately, this computational burden limits the capability of current models to elucidate critical calcium-dependent mechanisms associated with plasticity, learning, and memory, which emerge from network level activity.

To adequately model calcium-influenced cellular and network level behaviors, it is critical to construct a model that can efficiently and accurately replicate nonlinear calcium dynamics based on the numerous calcium processes on multiple levels, starting at the spine.

The focus of this article is to present a model describing the calcium dynamics at the postsynaptic spine of a CA1 pyramidal cell, as well as a methodology to adapt the previously defined model for multi-scale simulations. The calcium model presented aims to (1) consider the variety of subcellular processes that influence calcium at the spine, and (2) enable multiscale simulations that include the influence of said subcellular processes on calcium dynamics on a larger scale. In (1), we develop a mechanistic model that consists of various kinetic state models of receptor channels and pumps pertaining to calcium regulation at the spine; the mechanistic model is validated with experimental data from the literature and is used to study the subcellular processes involved in spine calcium dynamics. In (2), we implement a "input-output" model using the Volterra functional series trained on the nonlinear calcium dynamics from the mechanistic model; the Volterra functional series is a nonlinear systems filter that has been adapted previously to successfully model dynamics of nonlinear systems with reduced computational cost (Marmarelis and Marmarelis, 1978; Bharathy et al., 2008; Song et al., 2009a,b;

Berger et al., 2010; Tu et al., 2012; Hu et al., 2015). We demonstrate that the IO model requires less time to simulate than the mechanistic model, while still reproducing the complex nonlinear dynamics that arise from calcium interactions at the spine. Thus, we propose that the IO model is better suited for multi-scale modeling of calcium. In future studies, this will allow us to investigate the effects of how the various subcellular mechanisms in which affect spine calcium dynamics can influence cell to cell interactions at the cellular and network levels.

# MATERIALS AND METHODS

Here we first describe the models and parameters used for the mechanistic model; an overview diagram is presented in **Figure 2**, and a summary of the parameters is provided in **Table 1**. A precursor of the mechanistic model had been described previously in a conference publication (Hu et al., 2016), which was at the time an incomplete; this current model now describes the model in full with additional mechanisms, optimized parameters, and validation that is described in the results section. The development of the Input-Output model is described afterwards.

### Mechanistic Model

The premise of the mechanistic model is to build a physiological representation of the postsynaptic spine taking into account the components that are known to significantly influence spine calcium dynamics. These components are identified based on a number of experiments, reviews, and models found in the literature (Sabatini et al., 2002; Bloodgood and Sabatini, 2007; Higley and Sabatini, 2012; Bartol et al., 2015). In the mechanistic model, influx and efflux components such as the calcium channels (NMDAr, VDCC) and calcium pumps (PMCA, NCX) are represented as calcium current sources which can add or remove calcium in the spine compartment. As calcium ions flow in and out of the spine, the concentration is determined by calculating the change in calcium divided by spine volume—the standard definition of concentration. Buffers and intracellular calcium stores interact directly with the calcium within the spine using reaction rate equations. A schematic diagram of the interactions between the components in the model is presented in **Figure 3A**.

Besides modeling calcium in itself, each synapse also provides postsynaptic current to the model neuron. The postsynaptic current influences the postsynaptic potential, which later influences spine calcium via voltage-dependent calcium mechanisms. The postsynaptic current is governed by both AMPA receptor channel and NMDA receptor channel kinetic rate models. The NMDA receptor channel kinetic rate model is the same model described in this Supplementary Figure 1. The AMPA receptor channel rate model is the model described in Robert and Howe (2003). These components are part of the synaptic platform which

FIGURE 2 | Postsynaptic calcium can be influenced by a number of factors. Our focus is on calcium dynamics at the postsynaptic spine, as was highlighted in the red dashed rectangle in Figure 1. Models for receptor channels and pumps that contribute to calcium dynamics have been developed in several studies. In our model, these sources are integrated into a single platform to construct a mechanistic model of postsynaptic calcium dynamics.

TABLE 1 | List of the parameters, specifications, and models used in the mechanistic model and their sources.


has been developed in our lab and further details can be seen in Bouteiller et al. (2011), Allam et al. (2015), and Hu et al. (2015).

#### Spine Volume and Diffusion

The concentration of any constituent depends on the volume of its compartment. Spines come in a variety of shapes and sizes, with different forms such as thin spines, stubby spines, and mushroom spines (Stewart et al., 2005). The postsynaptic calcium model presented here considers the composition of an average mushroom spine of a CA1 neuron, with a spine volume of 0.1 µm<sup>3</sup> and the postsynaptic density volume set to 0.0032 µm<sup>3</sup> (Stewart et al., 2005). Here, we model the postsynaptic density as separate compartment due to the rapid increase in local calcium concentration during ion channel activation; the calcium concentration then diffuses into the rest of the spine. Diffusion from the postsynaptic density (PSD) to the spine compartment was approximated using the spine diffusion

Frontiers in Computational Neuroscience | www.frontiersin.org

Supplementary Table 6.

model described by Naoki et al. (2005). The diffusion rate is modeled as:

$$\left(\frac{d\left[Ca^{2+}\right]}{dt}\right)\_{\text{diffusion}} = D \frac{A}{d^{\*}V\_{spin}} (\left[Ca^{2+}\right]\_{\text{PSD}} - \left[Ca^{2+}\right])$$

(1)

$$\frac{d\left[\text{Ca}^{2+}\right]\_{\text{PSD}}}{dt} = -\frac{I\_{\text{N/DDA}}}{2FV\_{\text{PSD}}} - D \frac{A}{d^\*V\_{\text{PSD}}} (\left[\text{Ca}^{2+}\right]\_{\text{PSD}} - \left[\text{Ca}^{2+}\right]) \tag{2}$$

where A is the surface area between the PSD and the rest of the spine volume, d is the distance between the midpoints of the PSD and the spine, Vspine is the total volume of the spine, and - Ca2<sup>+</sup> PSD and - Ca2<sup>+</sup> Spine is the calcium concentration at the postsynaptic density and the spine, respectively. The postsynaptic density is dependent on INMDA, which is the calcium influx current from NMDA receptors. The equation for INMDA is described in Equation (3). The influx depends on Faraday's constant F and the volume of the postsynaptic density VPSD.

In our model, we consider the spine to be an isolated compartment from the rest of the cell, where the calcium does not flow out of the spine into the dendrite or vice versa. We make this assumption based on experimental evidence that spines are isolated electrically (Grunditz et al., 2008), and that less than 0.01% of the total calcium flux into the spine comes from the adjacent dendrite (Sabatini et al., 2002). As such, the model described here assumes that calcium exchange with the dendrite is insignificant and calcium dynamics from our spine model does not influence dendritic potential in our neuron model. We are aware that while dendritic calcium can be influenced by spine calcium dynamics, but in the current study, the focus of our model considers only the spine calcium and not dendrite calcium dynamics, as highlighted specifically as the red area in **Figure 1**. Future studies are planned that will expand our model beyond the spine level and integrate calcium across more hierarchies that had been presented in **Figure 1**. In the meanwhile, dendritic potential can influence spine calcium influx dynamics, as described in the following section.

#### Calcium Influx

Here, we first describe the models for NMDAr and VDCC, then explain protocols to simulate calcium influx mechanisms and activate the aformentioned models.

The calcium current contribution of the NMDA channel is calculated as:

$$I\_{\rm NMDA} = \mathfrak{g}\_{total \rm NMDA} \times V\_{\rm Ca} \tag{3}$$

$$\mathfrak{g}\_{total\text{NMDA}} = \mathfrak{g}\_{\text{NMDA}} \times \mathfrak{n}\_{\text{NMDA}} \tag{4}$$

INMDA indicates total calcium current that flows into the spine, gNMDA is the NMDA conductance to be described in the following paragraph and nNMDA represents the number of NMDA receptors with the given PSD volume and is set to 20 (Racca et al., 2000). VCa represents the calcium potential difference between intracellular and extracellular calcium, which again will be described in more detail later in the text.

The NMDA receptor channel is represented as a kinetic states model as described in Erreger et al. (2005). The model consists of 8 states that represent the resting, activation, opening, and desensitization states of the NMDA receptor channel. The magnesium ion blockade of the NMDAr channel also must be considered. To do so, the magnesium ion binding properties in the channel pore are described by:

$$\mathbf{g}\_0 = \mathbf{g}\_1 + \frac{\mathbf{g}\_2 - \mathbf{g}\_1}{1 + e^{\alpha V}} \tag{5}$$

$$g\_{\text{max}} = \frac{\text{go}}{1 + \left(\frac{[Mg^{2+}]\_{\text{o}}}{K\_{\text{0}}}\right)e^{-\delta zFV/RT}}\tag{6}$$

$$\mathbf{g}\_{\text{NMDA}} = \mathbf{g}\_{\text{max}} \times \mathbf{O}(t) \tag{7}$$

Where g<sup>0</sup> represents the total conductance in the absence of any magnesium, g<sup>1</sup> and g<sup>2</sup> represent the open state conductances with one glutamate bound and two glutamate molecules bound, respectively. g1is set at 40 pS while g<sup>2</sup> is set at 247 pS. The value <sup>α</sup> <sup>=</sup> 0.01 represents the steepness of the transition between <sup>g</sup>1and g2. - Mg2<sup>+</sup> o represents the external magnesium concentration and is set at 1 mM. K<sup>0</sup> is the equilibrium constant for magnesium set at 3.57, F is Faraday's Constant (9.64867.104 C mol−<sup>1</sup> ), R is the molecular gas constant (8.31434 J mol−<sup>1</sup> K −1 ), z is the valency of the calcium ion (2), and T is the temperature at 299.5 K. V represents the membrane potential. δ is the affinity between NMDAr and magnesium, which is dependent on the postsynaptic potential of the synapse; the value is set to 0.8. The variable O(t) is the open state probability governed by the kinetic rate equations for the NMDA model. The kinetic schema of the NMDAr model is presented in Supplementary Figure 1 where O (t)is represented as "Open." The rate constants which govern NMDA kinetics are presented in Supplementary Table 1. In the kinetic model, presynaptic action potentials cause vesicle glutamate release. Thus, in our model, a presynaptic event correlates to a glutamate spike in the NMDAr kinetic states model, moving the NMDAr channel kinetics away from resting state and causing them to open. For a more extensive description of the kinetic NMDAr model, please refer to Erreger et al. (2005).

NMDAr is differentially permeable to different ions when activated. Burnashev et al. (1995) reported that on average, calcium constitutes about 11% of the total ion current when NMDAr channels are opened. However, considering NMDA current alone lead to an incorrect representation of NMDAr mediated calcium influx, since the reversal potential of calcium (+ 50 mV) is considerably different from the reversal potential of NMDAr conductance (+0 mV). Therefore, in this model we calculate the influx using the Nernst equation, which instead depends on the difference between intracellular and extracellular calcium concentration. The Nernst equation used is as follows:

$$V\_{Ca} = -\frac{RT}{2F} \log\left(\frac{[Ca^{2+}]\_o}{[Ca^{2+}]}\right) \tag{8}$$

VCa represents the potential difference for calcium. R, T, and F represent the molecular gas constant, temperature, and Faraday's constant with the same values mentioned previously. [Ca2+]<sup>o</sup> is the extracellular calcium set as a constant concentration of 2 mM, while [Ca2+] represents intracellular calcium concentration at the spine. Here, we assume the calcium at the spine dominates the driving force represented by the Nernst potential. It is also possible to use the calcium concentration at the postsynaptic density to calculate the potential, but due to its small volume, minor fluctuations in current can result in drastic changes in the concentration value, which would lead to more erratic changes in the potential. Therefore, we consider the use of the overall spine concentration to be a more adequate representation of the Nernst potential for calcium.

The Nernst potential is typically used to determine the potential when the spine is at during the resting state. However, for our purposes we use the Nernst potential as an estimate of the driving force for calcium influx. Our justification is thus: In our spine model, we assume that all ions besides calcium remain constant and the electrochemical force for all other ions is zero. As mentioned earlier, considering only a percentage of synaptic current influx as calcium is inaccurate, since the reversal potential between calcium is much higher than the reversal potential of synaptic current. This case is true even for the Goldman–Hodgkin–Katz flux equation, since it depends on the membrane potential—if the membrane potential moves from negative to positive, the flux is also reversed. On the other hand, using a constant value for calcium potential, i.e., when calcium is at rest at 50 nM, does not account for changes in flux induced by increased calcium levels in the spine. Therefore, the use of the Nernst potential is an estimation of the electrochemical gradient when calcium concentration is changed. The concentration of intracellular calcium is orders of magnitude lower than the concentration of extracellular calcium, so the change is not large, but still significant enough such that we believe it should be accounted for, i.e., the value of VCa when intracellular calcium is 50 nM is approximately −60 mV, but can reach −30 mV when intracellular calcium reaches at 10µM.

Voltage dependent calcium channels (VDCCs) let calcium into the spine when there is a significant difference in membrane potential, such as action potential backpropagation from the postsynaptic neuron (Higley and Sabatini, 2012). Various types of VDCCs exist—each having different channel properties, mechanics, and functional roles—and are found on different types of cells (Catterall, 2011). For CA1 pyramidal cells, experimental evidence suggests that T-type VDCCs contribute the most to overall calcium concentration within dendritic spines (Bloodgood and Sabatini, 2007). It should be noted that L type and R type VDCCs are also present. However, the calcium influx contribution of L-types and Rtypes to the overall calcium concentration within spines was found to be insignificant. Instead, these channels tend to be concentrated into microdomains and activate secondary messenger pathways (Higley and Sabatini, 2012). Consequently, we consider the VDCC influx through T-type channels only.

The more specific details of the VDCC model are described in Supplementary Table 2. In general, the calcium contribution of the VDCC channel is calculated as:

$$I\_{\rm VDCC} = \lg\_{\rm VDC} \cdot m^2 \cdot h \cdot f\_{\rm drive} \tag{9}$$

Where IVDCC is the calcium current from the voltage dependent calcium channel. m and h are part of the Hodgkin-Huxley equation, with parameters as defined from Jaffe et al. (1994). fdrive is the driving force of the internal and external calcium dynamics, considered through modifications to the Hodgkin Huxley equation as reported by Poirazi et al. (2003) (equivalent to dvf in their model). Once again, the equations and parameters used in the model related to VDCC are presented in Supplementary Table 2. gVDCC is the he single channel conductance for VDCC is set to be 7.5 pS (Perez-Reyes et al., 1998) and the average number of VDCCs opened for each AP-evoked transient is 5 (Sabatini and Svoboda, 2000).

#### Calcium Efflux

Experimental evidence indicates that calcium is removed from the intracellular space through pumps and exchangers, but mechanistic details concerning calcium efflux at the postsynaptic spine are not yet fully understood. Generally, Plasma Membrane Calcium Pumps (PMCA) and Sodium-Calcium Exchangers (NCX) are the two prominent elements that participate in calcium extrusion in spines and small dendrites of CA1 pyramidal cell neurons (Scheuss et al., 2006). Overall it is thought that the constant active pumping by PMCA helps maintain the standard basal levels of calcium at ∼50 nM (Carafoli, 1991), while NCX helps to quickly extrude calcium in a short amount of time, such as during an action potential (Carafoli et al., 2001). One source in the literature suggests the PMCA isoform is type PMCA2w, although details on the dynamics and extrusion rates of the isoform are lacking (Burette et al., 2010). Details on NCX at spines are even less studied, with only one source indicating that NCX is present in larger numbers in dendritic shafts than in spines, though exact numbers are unknown (Lörincz et al., 2007).

Previously published models of spine calcium use calcium extrusion models as a calibration factor to fit simulations to experimental results (Naoki et al., 2005; Bartol et al., 2015). For our model, we are interested in using more physiologically accurate representations of extrusion, but such models specifically relating to the extrusion pumps and channels in spines are currently absent from the literature. Therefore, we have decided to use extrusion models from models of other physiological systems (Gall and Susa, 1999; Fridlyand et al., 2003), and adjust parameters according to the geometry of the synapse based on the surface area density of the models presented within these papers. The adjusted parameters are detailed in the following paragraph.

Both PMCA and NCX are represented through the Hill equation, which is typically used to describe binding properties of a ligand to the receptor:

$$I = I\_{MAX}\frac{\left[Ca^{2+}\right]^{h\_c}}{K^{h\_c} + \left[Ca^{2+}\right]^{h\_c}}\tag{10}$$

Here, IMAX is the maximum calcium current extruded from the model, K is the equilibrium constant and h<sup>c</sup> is the hill coefficient. For PMCA model parameters, the Hill coefficient is set to 2 and an equilibrium constant of 1µM (Fridlyand et al., 2003). The maximum calcium current was optimized through gradient descent based on experimental protocols highlighted in the results section; the optimized maximal PMCA current was set to be 0.25 pA. The NCX model has a Hill coefficient of 5, equilibrium constant value of 1.5µM and a conductance of 0.0117 pS (Gall and Susa, 1999); maximal current is then calculated from the sodium/calcium gradient and the given conductance. The higher Hill coefficient represented in the NCX model indicates a higher affinity for calcium ions when more calcium is bound. A table describing the parameter values is presented in **Table 2**.

Additionally, the NCX model accounts for the sodium/calcium gradient in which three sodium ions are exchanged for one calcium ion (Fridlyand et al., 2003):

$$V\_{Na,Ca} = \frac{RT}{F} \text{(3ln( } \frac{[Na^{+}]\_o}{[Na^{+}]\_i} \text{)} - \ln\left(\frac{[Ca^{2+}]\_o}{[Ca^{2+}]\_i}\right) \tag{11}$$

R, T, and F represent the molecular gas constant, temperature, and Faraday's constant with the same values mentioned previously. IMAX for NCX is determined based on the gradient difference. In the current model, intracellular and extracellular sodium concentrations are considered constant at 10 and 140 mM respectively. Similar to VCa from the Nernst equation, VNa,Ca is an estimate of the driving force to consider how calcium efflux will change depending on calcium concentration, providing a slightly better estimate compared to keeping VNa,Ca constant.

#### Buffers and Intracellular Calcium Stores

Buffering is an integral part of calcium dynamics at the postsynaptic spine, as up to 95% of the total intracellular calcium is bound to buffers (Sabatini et al., 2002). There are numerous types of buffers that can bind calcium; in our model, we specify two types of buffers, calmodulin and calbindin, while the other possible buffers are represented as a collection of generic calcium binding proteins (CBP). In our platform, the buffer models directly influence the free calcium concentration in the spine using reaction rate equations. The kinetic schemas and descriptions of the parameters and equations are presented in Supplementary Figure 2, **Table 1**, and Supplementary Table 3. Calmodulin is a ubiquitous calcium buffer which plays a role in AMPA receptor upregulation and synaptic potentiation when found in the postsynaptic spine. The calmodulin buffering parameters are defined in accordance to Zhabotinsky et al. (2006). Calbindin is a binding protein with four calcium binding


sites; here it is defined as a 9 states kinetic model, with parameters defined in the calcium model by Bartol et al. (2015). The CBP were calibrated after the previous two buffers were implemented, where the total buffered calcium at steady state reaches approximately 95% in the presence of all three buffers.

Intracellular calcium stores are known to play a large role in calcium dynamics, but current evidence on its impact particularly on dendritic spines in CA1 neurons remains controversial. So far it is found that approximately 19% of the total spine count contain endoplasmic reticulum (ER), with a majority of the ER-containing spines having a larger volume than others (approx. 0.06 µm<sup>3</sup> ) (Holbro et al., 2009). The ER apparatus within the spine was shown to have no IP3 receptors present while retaining a number of ryanodine receptors (Paula-Lima et al., 2014). As such, we have included in our model state representations for SERCA pumps and ryanodine receptors, but omit IP3 receptors. Just like the buffer models, the models pertaining to the intracellular calcium stores directly interact with the free calcium concentration in the spine. We describe the kinetic schema, parameters, and equations of SERCA and ryanodine receptors in Supplementary Figure 3, Supplementary Tables 4, 5. The SERCA pump is a 2 states model with equations and parameters derived from Higgins et al. (2006). We also included the ryanodine receptors model proposed by Williams et al. (2011).

#### Inputs Into the Mechanistic Spine Calcium Model

In **Figure 3A**, we provide a diagram of the interactions in the mechanistic model and describe the components which can influence calcium activity. Input stimulation predominantly occurs in two ways: (1) through synaptically activated transients, where presynaptic release of neurotransmitter activates the NMDA receptor channels on the postsynaptic density; and (2) AP-evoked transients, where stimulation of the postsynaptic neuron triggers an action potential, which is then backpropagated to the spines. Simulation of (1) is represented through presynaptic event-based activation of the NMDA receptor model, where a single event triggers the opening kinetics of the NMDAr model. The protocol for (2) is slightly more complex: in order to simulate AP-evoked transients in our model, we stimulate a number of synapses on the neuron model to invoke an action potential. Calcium concentration can then be measured on a stimulated or non-stimulated synapse, where the backpropagating action potential opens VDCCs and prompts calcium entry into the spine.

#### Input-Output Model

The development of the Input-Output (IO) model for postsynaptic calcium dynamics follows a protocol similar to the IO models that had been covered in Berger et al. (2010) and Hu et al. (2015). To describe briefly, the IO model uses the Volterra functional power series, with Laguerre functions as the basis equations of the Volterra series. The single input, single output (SISO) model of the Volterra functional series and the Laguerre equations have been previously described in Hu et al. (2015). In brief, we propose to use a simplified functional representation of the system under consideration.

In the current work, a multi-input, single output (MISO) model was developed with the inputs being the two major sources of calcium influx for the calcium model: (1) postsynaptic potential (V), and (2) NMDAr channel conductance (gtotalNMDA) based on glutamate-based calcium influx. One notable difference is that we herein consider the inputs to be continuous. Although we use spike trains as input to our synapse platform, the conductance and potential which are calculated from the spike trains are continuous. The use of continuous inputs contrasts with IO models that have been published previously assumed the inputs to be of binary nature (Hu et al., 2015). A pictorial representation of the mechanistic model and IO model is shown in **Figure 3B**, to highlight which parameters the two models have in common.

The inputs of the IO model were chosen because they represent outside activity that drives changes in calcium dynamics. The first input, the postsynaptic potential, is a factor known to drive calcium influx due to its effects on voltage dependent mechanisms (NMDAr associated channel and calcium channels). In both mechanistic and IO models, the postsynaptic potential depends on the neuron cell model. In our simulations, the Izhikevich model (Izhikevich, 2003) is used for most simulations except simulations that involve distance measurements, where the Migliore neuron model is used instead (Migliore and Migliore, 2012). Since the Izhikevich model has no geometry component, measurements with respect to distance cannot be performed using the Izhikevich model. The Migliore model, on the other hand, is a reconstruction of a hippocampal CA1 neuron, which consequently has dendritic geometry that can be used for our simulation study. In the neuron model, we simulate synaptic activity which results in firing of the cell; the action potential is then backpropagated to the spine model, which we use as input to either the mechanistic or the IO model. The second input, i.e., NMDA-R channel conductance, is a critical measure of pre- and postsynaptic activity; we determine the total NMDAr conductance based on both pre- and post-synaptic activity of the spine.

In our platform, opening of the NMDA receptor channel model: (1) Allows calcium influx into the spine, and (2) produces postsynaptic currents, which are passed to the neuron model and influences the postsynaptic potential. We chose the NMDAr channel conductance as an input parameter to the input-output model to account for the calcium influx while still allowing NMDAr channels to influence postsynaptic activity. The output response of the IO model is calcium concentration, calibrated using the calcium concentration obtained with the mechanistic model. Thus, the influence on calcium dynamics of all other components besides the NMDAr model are captured in the IO model. The use of the IO model then allows us to model complex nonlinear calcium dynamics without requiring the large number of components that would otherwise be necessary when using a mechanistic model.

To describe the structure of the IO model, we first begin with a description of the SISO Volterra series:

$$\mu\_{\rm SISO}(t) = \ c\_0 + \sum\_{j=1}^{L} c\_1 \begin{pmatrix} j \end{pmatrix} \nu\_j(t) + \sum\_{j\_1=1}^{L} \sum\_{j\_2}^{j\_1} c\_{2s} \begin{pmatrix} j\_1, j\_2 \end{pmatrix} \nu\_{j\_1}(t) \ \nu\_{j2}(t)$$

$$+\sum\_{j\_1=1}^{L} \sum\_{j\_2=1}^{j\_1} \sum\_{j\_3=1}^{j\_2} c\_{3s} \begin{pmatrix} j\_1, j\_2, \ j\_3 \end{pmatrix} \boldsymbol{\nu}\_{j\_1}(t) \ \boldsymbol{\nu}\_{j2}(t) \boldsymbol{\nu}\_{j3}(t) \tag{12}$$

$$\nu\_{\dot{\jmath}}(t) = \sum\_{\tau=0}^{M} b\_{\dot{\jmath}}(\tau) \, \varkappa(t-\tau) \tag{13}$$

Where uSISO is the single input Volterra series up to 3rd order. L refers to the total Laguerre functions in the SISO model, and c0, c1, c2<sup>s</sup> , and c3<sup>s</sup> refer to the coefficients associated with the 0th, 1st, 2nd, and 3rd order response of the series, respectively. v<sup>j</sup> (t) is the convolution of the input to the IO model with the basis function; x(t−τ ) is the input to the IO model; and b<sup>j</sup> (τ )is the j-th basis function. M is the memory window of the IO model, set to 5 s. The coefficients are determined during the training process to best fit the nonlinear response of calcium dynamics. For our case, we use the Laguerre basis functions for our model. The Laguerre equations are used for their orthogonality and convergence properties, which are characteristic of many biophysiological systems (Berger and Song, 2010; Ghaderi et al., 2011).

The structure of the Volterra functional series in the MISO model is similar to the SISO model. In the case of two inputs, the series consists of the summation of two SISO model components that account for two different inputs, then adding a cross-kernel component accounting for possible nonlinear interactions that may occur due to the presence of two inputs. The equations become thus:

$$\begin{aligned} u\_{\text{MISO}}(t) &= \left. c\_0 + u\_1(t) + u\_2(t) \right. \\ &\quad + \sum\_{k\_1=1}^{L\_1} \sum\_{k\_2=1}^{L\_2} c\_{2r} \left( k\_1, k\_2 \right) \nu\_{k1}^{\mu\_1}(t) \, \nu\_{k2}^{\mu\_2}(t) \\ &\quad + \sum\_{k\_1=1}^{L\_1} \sum\_{k\_2=1}^{L\_2} \sum\_{k\_3=1}^{L\_1} c\_{3r\_1} \left( k\_1, k\_2, k\_3 \right) \nu\_{k1}^{\mu\_1}(t) \, \nu\_{k2}^{\mu\_2}(t) \, \nu\_{k3}^{\mu\_1}(t) \\ &\quad + \sum\_{k\_1=1}^{L\_1} \sum\_{k\_2=1}^{L\_2} \sum\_{k\_4=1}^{L\_2} c\_{3r\_2} \left( k\_1, k\_2, k\_4 \right) \nu\_{k1}^{\mu\_1}(t) \, \nu\_{k2}^{\mu\_2}(t) \nu\_{k4}^{\mu\_2}(t) \end{aligned} \tag{14}$$

$$\nu\_j^{\mu\_i}(t) = \sum\_{\mathfrak{r}=0}^M b\_j(\mathfrak{r}) \, \mathfrak{x}^{\mu\_i}(t-\mathfrak{r}) \tag{15}$$

Here, uMISO is the equation for the multi-input Volterra series, and u<sup>1</sup> and u<sup>2</sup> are the single input Volterra series based on uSISO with the inputs the postsynaptic potential (V), and the NMDA receptor conductance (gtotalNMDA), respectively. The cross kernel components involve the number of Laguerre functions L<sup>1</sup> and L<sup>2</sup> from the first and second SISO components, respectively, and at all orders. v u k (t) is the associated basis functions convolved with inputs: postsynaptic potential (V), and the NMDA receptor conductance (gtotalNMDA). c<sup>r</sup> is then the associated coefficients for these terms. As a result, the nonlinearities influenced by having two different inputs are considered.

The IO model must be trained to tune its parameters and minimize the error with respect to the original system—in this case, the mechanistic model. The training process consisted in using the response of the original mechanistic model to random Poisson train events given at both low average frequency (2 Hz) and high average frequency (10 Hz), with a total of 1,000 events to ensure a wide range of activity. The low average frequency of 2 Hz was chosen to reflect the typical firing rate of hippocampal CA1 and CA3 neurons during resting state (Berger et al., 1988). Meanwhile, the high average frequency of 10 Hz is used to account for situations with higher levels of neuron activity in physiological conditions (Ranck, 1973). The coefficients associated with the basis functions were determined through the pseudoinverse matrix multiplication, and the optimal decay values associated with the Volterra series were estimated using gradient descent to find the parameters that resulted in the lowest root mean square error value in the comparison between the estimated and the actual result from the original model.

#### Platforms and Computational Tools

The mechanistic calcium model is described in the Systems Biology Markup Language (SBML) and was constructed using CellDesigner, a visual diagram editor for SBML (Funahashi et al., 2008). Our models were then run in MEMORY, a pythonbased platform designed for synapse-based simulations using the libSBML library and the simulation engine libroadrunner (Somogyi et al., 2015). In our studies which involve measuring calcium and dendritic potential based on distance from the soma, we required a neuron model with a detailed dendritic structure where synapses can be placed along the dendritic arbor. To achieve this, we use the CA1 pyramidal cell model designed by Migliore (Migliore and Migliore, 2012) simulated within NEURON cell simulation platform (Carnevale and Hines, 2006). The IO calcium model first required the coefficients to be determined, which were calculated within MATLAB; afterwards, the model was implemented into the MEMORY platform to run for simulation. Unless otherwise indicated, the input stimulus provided to synapses during simulation were Poisson 2 Hz randomized input trains, where each synapse was provided a unique randomized input. All simulations were conducted on a computer using the Fedora OS, with Intel quad-core 2.67 GHz processor and 8 Gb RAM.

# RESULTS

# Calcium Dynamics Calibration and Validation With Published Experimental Data

In the resting phase, the average cytosolic calcium concentration in spines is typically kept at around ∼50 nM, maintained by various pumps and buffers (Higley and Sabatini, 2012). Calcium influx occurs during activation of the various channels present on the spine. There are two major sources of calcium influx: Glutamate-dependent calcium influx, where calcium flows into the spine via NMDA receptor channels, and Voltage-dependent calcium influx, primarily through voltage-dependent calcium channels. For calibration of the mechanistic calcium model at the spine, we consider two scenarios: (1) calcium influx due to presynaptic activation, where the presynaptic terminal releases glutamate in response to a presynaptic action potential, and (2) calcium levels when the postsynaptic neuron is fired, leading to a backpropagating action potential (bAP). The response in (1) has been measured both in the presence and absence of postsynaptic depolarization (Sabatini et al., 2002; Higley and Sabatini, 2012). Our calcium model has a response of approximately 9 and 0.9µM in the presence and absence of postsynaptic depolarization, respectively, which is in line with the experimental data presented in the literature (**Figure 4A**). Similarly, calcium levels in response to bAP were simulated and compared to calcium levels measured in Sabatini et al. (2002) (**Figure 4B**). The backpropagation factor assumed that the spine was a distance of <sup>∼</sup>150µm from the soma, similar to the synapses recorded in the literature. The simulated results showed an amplitude of approximately 700 nM, comparable to measurements from Sabatini et al. (2002).

# Calcium Fluctuations Vary According to the Inter-spike Intervals Between Presynaptic and Postsynaptic Activity of the Neuron

Due to the number of mechanisms in place, the integration of glutamate-dependent calcium influx and voltage-dependent calcium influx leads to further complex dynamics in calcium concentration at the spine. When the membrane potential increases as a result of postsynaptic events, the magnesium block is removed in the NMDA receptor channel pore, increasing the influx of ions when the receptor is activated (Jahr and Stevens, 1990; Ambert et al., 2010). In our model, we measured the maximum amplitude of calcium concentration at varying prepost intervals to determine its influence on calcium dynamics in the spine (**Figure 5**). Through the study, we found that the highest calcium concentration peak reached was 9.88µM, when the postsynaptic event occurred 3 ms after the presynaptic event. It was also noted that for pre-post intervals, where the presynaptic event precedes the postsynaptic event, there is a notable increase in the magnitude in comparison to the calcium response when there is only presynaptic activation with no postsynaptic activation. The increase in maximum calcium amplitude is present when the pre-post interval ranges from 0 to 80 ms; beyond 80 ms, there appears to be no significant change in maximum amplitude compared to presynaptic activation only. In the case of post-pre intervals, where the presynaptic event follows the postsynaptic event (corresponding to negative delays in **Figure 5**), calcium amplitude begins rising sharply starting at 20 ms all the way to 0 ms. It is interesting to note that while our model does not account for synaptic plasticity, the pre-post interval time-scale dependency of calcium amplitude on pre-post intervals resembles the well-known spike timing dependent plasticity (STDP) curve presented by Bi and Poo (2001), particularly in regards to synaptic strengthening when post-synaptic activation follows pre-synaptic activation. The correlation in timescale dependence between the STDP curve and the presented results from the mechanistic calcium model suggests the mechanistic model can be considered a plausible model basis for a plasticity model in the future.

# Postsynaptic Calcium Activity Depends on the Distance Between the Synapse and the Soma

The experimental studies conducted by Sabatini et al. (2002) observed synapses located approximately 150µm from the soma.

FIGURE 4 | Validation of single event responses of the mechanistic model of calcium concentration at the postsynaptic spine. (A) The response of the postsynaptic spine model to a single presynaptic event with an elicited action potential (in light blue, dashed) that reaches <sup>∼</sup>9µM, and without any resulting action potential (in blue, solid) that reaches <sup>∼</sup>1µM, in line with expected amplitudes that have been reported in experimental studies and reviews (Sabatini et al., 2002; Higley and Sabatini, 2012). Neurotransmitter release following a presynaptic event opens NMDA receptor channels, causing calcium influx; the degree of influx is also dependent on membrane potential due to the voltage-dependent magnesium block. (B) Response to a single postsynaptic backpropagation event by the postsynaptic spine model (in red) in comparison to reported calcium response seen in the study by Sabatini et al. (2002) (in black). Voltage-dependent calcium channels (VDCC) open in response to a change in calcium levels—the result shown here is the response due to low-voltage activated VDCC-T type channels in the postsynaptic spine model.

FIGURE 5 | The magnitude of the calcium response changes depending on the interval between presynaptic and postsynaptic events. Timing interval between presynaptic and postsynaptic events can influence the amplitude of calcium dynamics in the mechanistic model of calcium at the postsynaptic spine. Pre-Post interval is defined as the amount of time in milliseconds the postsynaptic event (presynaptic neuron activation, NT release is triggered) occurs after the presynaptic event (postsynaptic neuron activation, the action potential is backpropagated to the spine). Black dots indicate measured maximum calcium response for simulations with the designated pre-post interval. In the inset, A., B., and C. show the simulated calcium profile in response to different pre-post intervals. The change in maximum calcium amplitude reflects the kinetics of the NMDAr channel dynamics, where the voltage dependent magnesium block results in the nonlinear behavior of the calcium response.

Studies have shown that backpropagation signal properties of the synapse and dendrite can change depending on distance (Golding et al., 2001). To observe the influence of distance on the calcium response, we used the CA1 pyramidal neuron model by Migliore and Migliore (2012) to simulate 50 randomly placed synapses on the stratum radiatum, with the closest synapse having a distance of 72.62µm and the furthest synapse has a distance of 407.03µm; these values are close to the minimum and maximum range specified for the stratum radiatum (Megías et al., 2001). When synapses were randomly placed on the pyramidal cell model, we measured the dendritic diameter of the synapse locations. It was found that 45 (90%) of the synapse locations had a diameter of 0.5µm. This corresponds to the measured diameters of the thin dendrites located in the stratum radiatum in experimental findings (Megías et al., 2001). Other diameters were 2, 2, 1.2, 1.2, and 0.18µm. Our simulations from these synapses showed no influence of the diameter on our measured results. In our first set of simulations, the synapses were stimulated with a single presynaptic event, then a fixed single postsynaptic event following 10 ms afterwards. **Figure 6A** shows a schematic of the simulation setup. In **Figure 6B**, we summarize the results where for each synapse we consider (**Figure 6Bi**) the resting potential, **Figure 6Bii** maximum amplitude of the backpropagating action potential during stimulation with a pre-post interval of 10 ms, and **Figure 6Biii** maximum calcium response during stimulation with a pre-post interval of 10 ms. We note that both the resting potential from **Figure 6Bi** and the max bAP from **Figure 6Bii** are properties inherent to the neuron model described by Migliore and Migliore (2012). For **Figure 6Biii**, we simulate calcium

FIGURE 6 | measuring calcium levels for synapses located at different distances from the dendrite. Presynaptic stimulation indicates simulated presynaptic release on the respective synapse, while postsynaptic stimulation indicates simulation of the soma, inducing a backpropagating action potential. (B) Synapses were stimulated with a presynaptic release event at time 0, followed by a simulated current injection into the soma of the neuron 10 ms following the presynaptic event, triggering an action potential and invoking backpropagation. (i) The dendritic resting potential values from the 50 locations with respect to distance. On average, the potential is slightly higher in distal than in proximal locations by approximately 2 mV. (ii) In contrast to the baseline potential, the maximum potential reached with the backpropagation of a single postsynaptic spike becomes more attenuated when further away from the soma. (iii) The influence of the attenuation is seen in the calcium measurements, where maximum calcium is reduced when moving further away from the soma, with 300µm and beyond the backpropagation becomes negligible. (C) The pre-post interval was varied between simulations, and the maximum calcium response was measured for each simulation at each synapse location. The darker lines indicate calcium responses from synapse locations closer to the soma, while lighter lines are responses from locations further away from the soma. The blue dotted line is where the pre-post interval is 10 milliseconds, reflecting results that are seen in (Bi) through (Biii).

dynamics at the spine using our mechanistic calcium model described in this manuscript. In **Figure 6C**, we repeated the simulations with different pre-post intervals and measured the maximum calcium response at the spine of each synapse location.

Our simulation results indicated that baseline potential is slightly increased in distal dendrites than in proximal dendrites, with a range lying between −73.4 mV and −71 mV. Such a difference appears rather small, but still constitutes a notable trend with respect to distance. Differences up to 2 mV between somatic and dendritic resting potential have been observed experimentally, and our model falls in line within these constraints (Golding et al., 2001). Conversely, the maximum potential reached after pre-post event stimulations decreases with respect to distance; proximal dendrites are more likely to reach a higher maximum potential than distal synapses.

Observations of the calcium concentration levels at the postsynaptic spine indicated that at a pre-post interval of 10 ms, the amplitude of the calcium concentration peak in spines decreased as distance with the soma increased. Again, beyond 300µm we found that calcium amplitudes do not extend much beyond 1µM, likely due to the reduced influence of the postsynaptic activity. Subsequent simulations where the pre-post intervals are changed demonstrated that the influence of the timing between presynaptic and postsynaptic events is more prominent in proximal synapses than in distal synapses. In particular, there no longer appears to be any dependency of the max calcium response on the pre-post interval timing for the synapses farthest away from the soma. These simulations are in line with studies on the influence of distance on STDP, where it was found that backpropagation induces LTP more commonly in proximal synapses, while at distal synapses LTD occurs more frequently in response to the same backpropagating potential (Sjöström and Häusser, 2006).

Results presented here suggest that the bAP is significantly attenuated in distal spines to the degree that a single pre-post events does not trigger much calcium influx. At the time of writing, experimental data of spine calcium levels based on distance have not been documented in the literature. However, the results presented on backpropagation are in line with what has been observed in experimental studies, such as Golding et al. (2001), where it was observed the bAP amplitude is reduced when further away from the soma, and especially beyond 300µm. The reason for attenuation is likely due to two factors: (1) changes in active conductance with respect to distance from the soma, where at distal dendrites there is a higher density of potassium channels and low density of calcium and sodium channels (Bikbaev et al., 2016); and (2) the branching of the dendritic arbors, which has also been seen to contribute to the attenuation of the bAP (Golding et al., 2001). In distal synapses, there may be other mechanisms at play that may more prominently influence signaling and plasticity to compensate for attenuated bAP, such as modulation by glial cells and neurotransmitters (acetylcholine, brain derived neurotrophic factor, dopamine noradrenaline) (Edelmann et al., 2017).

# A Third Order, Multi-Input Input-Output Calcium Model Closely Replicates the Response of the Mechanistic Calcium Model at Lower Frequency

Each individual component in the mechanistic calcium spine model has its own degree of computational complexity, and the integration of all the components also compounds the overall computational burden. As a result, simulation of the calcium dynamics for a larger number of spines becomes increasingly difficult. We demonstrate here the use of an input-output model based on the Volterra functional series that reduces computational cost of simulating calcium dynamics. The output of the model is calcium concentration. The inputs to the proposed model are membrane potential and NMDA receptor conductance. The NMDA receptor model is the only component in the calcium dynamics model that we consider outside of the IO calcium model, since an IO NMDA model had been developed before and can be utilized in its place (Hu et al., 2015). In brief, the IO NMDA model is a single-input-single output model which uses the Volterra series with Laguerre basis functions to predict the open state probability of the NMDA receptor channel ("Open" from Supplementary Figure 1). The open state probability is then used to calculate the conductance based on the magnesium blockade equation (Equations 3–5). However, for the purposes of consistency, in our simulations we use the kinetic NMDA receptor model to properly compare results only between the IO calcium model and mechanistic calcium model. Training the IO model requires keeping track and replicating calcium concentration profiles from the mechanistic model in three separate conditions. First, when only presynaptic stimulation is applied; then, when only postsynaptic stimulation is applied (back-propagated action potential); and finally, when both presynaptic and postsynaptic stimulations are applied to the spine. For each type of stimulation, we used Poisson random interval trains for 1,000 events at 2 Hz and 1,000 events at 10 Hz for a total of 2,000 events, as each input. This gives us a total of 8,000 events for the model to be trained on: 4,000 events in total for presynaptic stimulation, and 4,000 events total for postsynaptic stimulation. The large number of events at lower and higher frequencies gives the IO model an adequate range of nonlinear dynamics to be trained on. At the end of the training phase, the root mean square (RMS) difference is calculated and normalized to the maximum value and minimum of mechanistic model response. The difference between the mechanistic model and the trained IO model was 6.79% with the given training data.

We then validated the trained IO model with a naive train of presynaptic and postsynaptic stimulations for both 2 and 10 Hz and compared the results with the ones obtained with mechanistic model. The trained IO model was found to be more accurate at lower frequencies: the validation error at 2 Hz was 8.15%, while the error for 10 Hz reached 16.9% (**Figure 7**). We have also provided a comparison between the mechanistic model response and the response from the linear calcium model by Shouval et al. (2002), which is presented in **Figure 8**. The linear calcium model is presented as:

$$\frac{d\left[Ca^{2+}\right]}{dt} = I\_{\text{NMDA}}\left(t\right) - \left(\frac{1}{\tau\_{Ca}}\left[Ca^{2+}\right]\right) \tag{16}$$


Following validation, the computational time to run the IO model and the mechanistic model was determined based on number of spines with 2 Hz Poisson random interval train inputs. We find that the IO model finished the simulations faster than the mechanistic model, where the runtime of the IO model required around half the time to finish a simulation compared to the mechanistic model (**Figure 9**).

Another advantage of the IO calcium model is that the framework of the input-output model is easily implemented and adaptable to other simulation platforms. To test the performance of the IO calcium model simulated as an embedded mechanism within the NEURON engine, we adapted the IO model into a module file for the NEURON platform and compared cell level simulations based on number of spine instances. Two types of models were simulated with different spine configurations: (1) spines using both an NMDA 8 state model and the IO calcium model, and (2) spines with only the NMDA 8 state model and no IO calcium model. Simulation protocols with 10, 100, 500, 1,000, 5,000, and 10,000 spine instances were conducted; simulations were run in fixed time step of 0.1 ms, a randomized poisson input train of 2 Hz frequency, and the overall simulated time period is for 20 s. Simulation times were then benchmarked to determine how much of a computational burden is added when including IO calcium model within spines. Simulations were repeated 10 times each to derive the standard deviation in the simulation time. Results are shown in **Figure 10**. Our simulations concluded that at 10,000 spines, the computational burden increases from 9.3 to 13.3 h. For details on the variation in the benchmarks, the standard deviation of the benchmarking data from **Figures 9**, **10** are plotted in Supplementary Figure 4. While the increased required simulation time is not insignificant, the IO framework still gives a viable option for simulating complex postsynaptic calcium dynamics on a larger scale—with numerous spines on a neuron or a neuron network.

# DISCUSSION

This article describes the development and simulation of a model of the postsynaptic calcium concentration in the spine. The model presented is an integration of various mechanisms which shape the dynamics of calcium concentration at the postsynaptic spine, comprising elements that contribute to calcium influx, calcium extrusion, and buffering. Experimental studies on spine signaling have focused on calcium more than any other signaling molecule within active spines (Higley and Sabatini, 2012), as calcium dynamics and its effectors (NMDA, VDCC, etc.) have been repeatedly shown to strongly influence plasticity and learning. Calcium has also been implicated as a role player in neurodegenerative diseases such as Alzheimer's Disease (Khachaturian, 1994; Alberdi et al., 2010). Our goal in modeling calcium is to: (1) explore mechanisms and details underlying calcium dynamics that would otherwise be difficult to achieve with experimental studies alone (i.e., influence of pre-post timing and distance on spine calcium, where the researcher must take multiple time measurements at multiple spine locations, would be difficult to measure in experimental setups), and (2) reduce computational complexity of the calcium model to enable multiscale simulations. We have presented a viable model which is supported by experimental data. We configured the model to replicate a particular type of synapse—a glutamatergic CA3-CA1

FIGURE 8 | Comparison between the linear calcium model (Shouval et al., 2002) and the mechanistic calcium model. (A,B) Shows the responses of the mechanistic model (in blue) and the linear calcium model (in orange) over the course of 20 s given Poisson randomized presynaptic and postsynaptic events with an average of 2 and 10 Hz, respectively. The root mean square difference between the two models is 80.52% for the 2 Hz average response, and 89.75% for the 10 Hz average response. (C,D) Shows a direct comparison between the calcium response values from the mechanistic model (x axis) and the linear calcium model (y axis) from the 2 and 10 Hz responses, respectably.

FIGURE 9 | Comparison between the Mechanistic and IO model runtime vs. number of spines. Each spine represents the kinetics for synaptic transmission as well as calcium dynamics, which is either represented by the mechanistic calcium model or the IO calcium model. Benchmarking was conducted at 5, 10, 15, 20, 30, 40, and 50 spines, and simulations were repeated 10 times to derive the standard deviation of the runtimes. For each simulation, a 2 Hz Poisson random interval train input was given to all synapses. Overall, the IO spine model required on average a little less than half the runtime needed to finish a simulation compared to the mechanistic model.

FIGURE 10 | Benchmarking the IO model within the NEURON simulation platform. The IO model was re-implemented as a module file usable within the NEURON platform and simulated on a compartmental neuron model for 20 simulated seconds at 0.1 ms timesteps. The number of spines was varied for each simulation ranging from 10 to 10,000, where a spine was defined as a kinetic NMDA 8 state model, either with or without the IO calcium model and the simulation runtime is plotted here. Each simulation was repeated 10 times each to derive the standard deviation, shown as error bars in the figure.

synapse of a pyramidal cell neuron; it incorporates the nonlinear dynamics that result from interactions between the components that contribute to spine calcium concentration.

Beyond experimental validation, we presented simulated results with the mechanistic model which show changes in spine calcium activity as a function of presynaptic and postsynaptic intervals—a standard protocol for inducing spiketiming dependent plasticity. In STDP, the weight of a synapse changes after repeated identical pre-post stimulations at given pre-post intervals. In hippocampal CA1 glutamatergic spines, intervals where presynaptic stimulation precedes postsynaptic stimulation induce synaptic potentiation, with the strength of the potentiation inversely proportional to the interval distance between the pre- and post-stimulation (Bi and Poo, 2001). Our model demonstrates that similarly, calcium influx is significantly amplified when presynaptic stimulation precedes postsynaptic stimulation, and that the amplitude is also inversely proportional to the interval size. Many plasticity associated signaling cascades are activated by calcium—for example, AMPA receptor upregulation into the spine is a known indicator for synaptic strengthening (Zhabotinsky et al., 2006). This process is initiated by spine calcium binding with CaMKII and triggering secondary messenger pathways. Likewise, recruitment of actin will lead larger spines—this, too has been associated with calcium interaction (Araya, 2014).

It is thought that the major influence on the calcium brought about by presynaptic/postsynaptic interactions is the NMDA receptor channel kinetics, but we have demonstrated in our simulations that the NMDAr channel alone is not the sole contributor of the nonlinear dynamics of calcium in the spine. The role of NMDAr in synaptic activity is considerably important: it has been shown experimentally that NMDA contributes to synaptic plasticity and LTP (Sakimura et al., 1995; Grover et al., 2009; Larson and Munkácsy, 2015). Many LTP models are based around this hypothesis, where NMDAr models are used to represent calcium influx, and repeated stimulation leads to calcium induced plasticity (Shouval et al., 2002; Standage et al., 2014). However, the NMDA representation in such models is a simple, linear representation where the NMDAbased calcium influx is represented as a ratio proportional to the bAP. This poorly reflects on nonlinear calcium dynamics in two ways: (1) the simplified version of NMDAr dynamics ignores important dynamical features that are known to be associated in NMDAr channels, such as desensitization (Mayer et al., 1989), which is included as a state in the NMDAr kinetic model used in our mechanistic calcium platform, and the magnesium blockade (Calabresi et al., 1992) which is instead roughly approximated using a BPAP curve; (2) there is no influence or contribution from other elements or properties from the spine, which can drastically alter the calcium response. Meanwhile, our model integrates validated channel kinetics within the confines of the spine compartment. Thus, our mechanistic model can consider the nonlinear aspects of calcium influx which are influenced by NMDAr channels, along with other channels, pumps, and buffers that regulate spine calcium concentration—all of which influence observed calcium levels at the spine. Especially to note is the spine volume and the buffers within the spine. Changes to volume can result in undercompensation (in larger volumes) or overcompensation (in smaller volumes) of calcium concentration, unless the mechanisms which govern calcium dynamics are scaled appropriately (O'Donnell et al., 2011). Furthermore, experimental evidence has shown that changes in volume also can result in change in AMPAr expression and upregulation (Noguchi et al., 2011), which is also important for synaptic plasticity. Such mechanisms revolved around volume are ignored in linear calcium concentration models, but can be studied in future simulations using the mechanistic model. Buffering provides another layer of complexity that will influence the amplitude and decay of the calcium response, yet there is no consideration for buffers in the linear calcium models. Our results demonstrate that, in the study of spine calcium, these components are important contributors to the nonlinear response of calcium.

The investigation into calcium dynamics at the postsynaptic spine is our latest study in the use of computational synaptic modeling platform to better understand synaptic activity and signaling: (1) We modeled the impact of astrocytic glutamate uptake (Allam et al., 2012) and ionotropic receptor distribution (Allam et al., 2015) on synaptic transmission in glutamatergic CA1 synapses; (2) Our synapse platform has been adapted to cellular and network levels in simulation (Bouteiller et al., 2011) to observe effects of nonlinear activity of synapses in a network simulation; (3) Large scale simulation models containing millions of neurons have also been developed within our research lab (Hendrickson et al., 2015); (4) and efforts had been made to adapt the complex nonlinear postsynaptic response of mechanistic synapses to large scale simulations using input-output modeling (Hu et al., 2015). Our modeling platform is consistently under expansion, with current projects considering the effects of modulators such as acetylcholine and how intracellular calcium stores influence metabolism and pump activity. The mechanistic and IO calcium model we describe in this manuscript expands our modeling platform to simulate not only synaptic transmission, but complex calcium dynamics as well. From here on, we plan to investigate and implement mechanisms that are based on spine calcium (the CaMKII signaling pathway, for example) and move to the next hierarchical level of calcium dynamics, the calcium response at the dendrite.

Expansion into large-scale, multi-scale modeling with complex biologically accurate synapse dynamics requires reduction of the computational burden while minimizing loss in accuracy. Spine calcium plays a key role in synaptic plasticity and influences communication between neurons, and understanding how calcium dynamics change network properties on a large scale can give us a better sense of the mechanisms that give rise to plasticity. Key downstream processes are influenced by the slightest changes in calcium dynamics (timing, magnitude, frequency, decay) (Evans and Blackwell, 2015). For example, an increase in spine calcium levels can activate signaling cascades that lead to either LTP or LTD induction (Lisman, 1989; Artola et al., 1990; Malenka and Bear, 2004); however, it is also observed in experiments that there is not a simple threshold that distinguishes when LTP or LTD occurs during calcium influx (Neveu and Zucker, 1996)—emphasizing even more the need of an integrated model of varying calcium dynamics, not just a linear model based on thresholds. Hence, we believe accurate representations of nonlinear calcium are required not only at the subcellular scale models, but on larger network-level models as well. Our model can provide an accurate reflection on the magnitude, duration, and location of spine calcium response—nonlinear dynamics that have been described to be more and more important for calcium based synaptic plasticity (Evans and Blackwell, 2015). Furthermore, neurodegenerative disease are often accompanied by an imbalance in calcium levels (Arundine and Tymianski, 2003). Nonlinear calcium models can be modified to represent pathological conditions, and multi-scale modeling can help identify the network level changes that occur with neurodegeneration and disease.

However, the computational cost of using many instances of the mechanistic calcium model in full exceeds the computational capacity of even the most recent high-performance computers; a method to improve computational efficiency is needed. Our previous work considers the use of the Volterra functional series to develop an IO model for the postsynaptic response to a synapse (Hu et al., 2015). Using the same input-output framework, we adapted this method to reduce the computational burden of modeling calcium dynamics. We have shown that using the IO model reduces the required simulation time by two to threefold compared to the mechanistic model within our MEMORY platform. We also demonstrate that the IO framework can be easily adapted into other platforms such as NEURON, where, in our setup, 10,000 instances of the IO calcium model with the NMDAr kinetic model can be simulated on a complex, morphological, compartmental cell model, resulting in 1 h additional simulation time compared to the same protocol using the NMDAr kinetic model but without the IO calcium model. However, the current IO model as described has a limitation that must be addressed: it is limited to a 3rd order model, with higher order models requiring exponentially increasing memory requisites. Spine calcium dynamics become progressively more nonlinear when given higher frequency input, such as high frequency stimulation protocols often used in LTP induction. As such, other IO model frameworks are being investigated, such as the Laguerre-Volterra network structure (Geng and Marmarelis, 2016), as possible enhancements leading to even more efficient computational modeling of complex dynamic systems.

The concept of using computer simulations to study postsynaptic calcium dynamics is not new. There have been several computational models of spine calcium that have been developed previously, and their work has provided useful insights on the dynamics and importance of calcium (Shouval et al., 2002; Standage et al., 2014; Bartol et al., 2015). Generally, current calcium models are divided into two categories: (1) Phenomenological models which describe very few mechanistic aspects of calcium dynamics, but help understand its influence on synaptic plasticity and LTP (Shouval et al., 2002; Naoki et al., 2005; Zhabotinsky et al., 2006; Standage et al., 2014); and (2) detailed, stochastic models which describe calcium all the way down to each individual ion (Bartol et al., 2015). In (1), the models do not extensively consider calcium dynamics at length and may even consider calcium as a linear system. Instead, models from (1) evaluate the downstream effects of calcium on important synaptic processes such as plasticity. In contrast, the detailed calcium model in (2) considers a complete reconstruction of a small area on the CA1 pyramidal cell neuron, including specific channel densities and exact volume and shape reconstruction of a 6 <sup>×</sup> <sup>6</sup> <sup>×</sup> <sup>5</sup> <sup>µ</sup>m<sup>3</sup> cube of neuropil in a Monte Carlo based stochastic simulator. However, such a model is difficult to adapt outside of the scope of the reconstructed area and is computationally intensive, making it unsuitable for larger scale simulations. The models in this article help bridge this discrepancy: (1) the mechanistic model is capable of replicating complex non-linear interactions between the elements that shape spine calcium dynamics, and (2) the input-output model provides a method to simulate these complex calcium dynamics on a larger scale.

The spine is a constantly changing organelle as a result of development, plasticity, and/or pathological conditions. The current spine calcium model as described here represents only a snapshot of a particular spine, constrained by static parameters based on the averaged responses from experimental data. Future renditions of our model will not be limited to a single type of synapse. Our spine calcium model has potential to be adapted to varying physiological states (i.e., different morphologies and channel distributions) and pathological conditions (such as Alzheimer's disease). Furthermore, the model has potential applications in drug discovery, for in silico testing of compounds that modulate calcium (either directly or via channel/pump interactions). The calcium model is an expansion of the synapse model framework that is constantly being built upon to provide

#### REFERENCES


extensive and detailed multi-level models that can help explore the pathways and processes of that take place in the spine and influence synaptic plasticity, neuron communication, and pathological processes.

# AVAILABILITY

Model scripts and code will be made available in the future on the EONS synaptic platform modeling site, www.synapticmodeling. com.

# AUTHOR CONTRIBUTIONS

EH and J-MB: concept and design of study; EH and AM: data acquisition; EH, DS, and J-MB: analysis and/or interpretation of data; EH and J-MB: drafting of the manuscript; CB, DS, J-MB, and TB: critical revision; EH, AM, CB, DS, J-MB, and TB: approval of the manuscript to be published.

#### ACKNOWLEDGMENTS

Work supported by NIBIB grant U01 GM104604 to TB.

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncom. 2018.00058/full#supplementary-material

calcium transients from individual proteins. Front. Synaptic Neurosci. 7:17. doi: 10.3389/fnsyn.2015.00017


located in dendritic spines. Neuron 53, 249–260. doi: 10.1016/j.neuron.2006. 12.017


Engineering in Medicine and Biology Society. Annual Conference (Boston, MA), 3581–3584.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Hu, Mergenthal, Bingham, Song, Bouteiller and Berger. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Dopamine Receptor Activation Is Required for GABAergic Spike Timing-Dependent Plasticity in Response to Complex Spike Pairing in the Ventral Tegmental Area

#### Ludovic D. Langlois, Matthieu Dacher † and Fereshteh S. Nugent\*

Department of Pharmacology, F. Edward Hebert School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, MD, United States

#### Edited by:

Vito Di Maio, Istituto di Scienze Applicate e Sistemi Intelligenti "Eduardo Caianiello" (ISASI), Italy

#### Reviewed by:

Fatuel Tecuapetla, Instituto de Fisiología Celular (IFC), Mexico Jose Bargas, Universidad Nacional Autónoma de México, Mexico

\*Correspondence:

Fereshteh S. Nugent fereshteh.nugent@usuhs.edu

#### †Present address:

Matthieu Dacher, Université Pierre et Marie Curie, Physiologie de l'Insecte, Signalisation et Communication—UMR 1272, Versailles, France

> Received: 20 July 2018 Accepted: 30 August 2018 Published: 21 September 2018

#### Citation:

Langlois LD, Dacher M and Nugent FS (2018) Dopamine Receptor Activation Is Required for GABAergic Spike Timing-Dependent Plasticity in Response to Complex Spike Pairing in the Ventral Tegmental Area. Front. Synaptic Neurosci. 10:32. doi: 10.3389/fnsyn.2018.00032 One of the most influential synaptic learning rules explored in the past decades is activity dependent spike-timing-dependent plasticity (STDP). In STDP, synapses are either potentiated or depressed based on the order of pre- and postsynaptic neuronal activation within narrow, milliseconds-long, time intervals. STDP is subject to neuromodulation by dopamine (DA), a potent neurotransmitter that significantly impacts synaptic plasticity and reward-related behavioral learning. Previously, we demonstrated that GABAergic synapses onto ventral tegmental area (VTA) DA neurons are able to express STDP (Kodangattil et al., 2013), however it is still unclear whether DA modulates inhibitory STDP in the VTA. Here, we used whole-cell recordings in rat midbrain slices to investigate whether DA D1-like and/or D2-like receptor (D1R/D2R) activation is required for induction of STDP in response to a complex pattern of spiking. We found that VTA but not Substantia nigra pars compact (SNc) DA neurons exhibit long-term depression (LTDGABA) in response to a combination of positive (pre-post) and negative (post-pre) timing of spiking (a complex STDP protocol). Blockade of either D1Rs or D2Rs prevented the induction of LTDGABA while activation of D1Rs did not affect the plasticity in response to this complex STDP protocol in VTA DA neurons. Our data suggest that this DA-dependent GABAergic STDP is selectively expressed at GABAergic synapses onto VTA DA neurons which could be targeted by drugs of abuse to mediate drug-induced modulation of DA signaling within the VTA, as well as in VTA-projection areas, thereby affecting reward-related learning and drug-associated memories.

Keywords: ventral tegmental area, VTA, spike-timing dependent plasticity, STDP, synaptic plasticity, long-term depression, LTD, GABAergic synapses

**Abbreviations:** AKAP, A kinase anchoring protein; CaN, Calcineurin; DA, dopamine; D1R, dopamine D1-like receptor; D2R, dopamine D2-like receptor; IPSC, Inhibitory postsynaptic current; LTD, long-term depression; LTP, Long-term potentiation; LFS, Low frequency stimulation; NAc, Nucleus accumbens; bAPs, propagating action potentials; PKA, protein kinase A; RMTg, rostromedial tegmental area; STDP, Spike-timing dependent plasticity; SNc, Substantia pars compacta; VTA, Ventral tegmental area.

# INTRODUCTION

In the past few decades, synaptic plasticity has emerged as an important candidate mechanism for drug-induced changes in reward-related neural circuits where addictive drugs usurp synaptic mechanisms underpinning reward/motivational or aversive behavioral learning to alter dopamine (DA) signaling from the ventral tegmental area (VTA), a critical brain region involved in reward and motivation (Pignatelli and Bonci, 2015; Langlois and Nugent, 2017). Drug-induced synaptic plasticity has attracted considerable interest in studies of drug addiction because strong and durable memories associated with drug experience are demonstrated to promote compulsive drug taking, craving and relapse.

Experimental synaptic plasticity can be induced using traditional induction paradigms as well as spike-timingdependent plasticity (STDP) protocols. STDP is considered a physiological form of plasticity that relies on the relative timing of neuronal activity (Feldman, 2012). Specifically, STDP at glutamatergic and GABAergic synapses within DA-related neural circuits likely presents a critical synaptic learning rule for encoding natural reward-related learning and memory (Pawlak et al., 2010; Langlois and Nugent, 2017; Foncelle et al., 2018). The Hebbian learning rules of STDP dictates that long-term potentiation (LTP) is induced when presynaptic activity precedes postsynaptic spiking (pre-post spiking, positive timing), whereas reversing the order induces long-term depression (LTD, post-pre spiking, negative timing; Dan and Poo, 2006; Caporale and Dan, 2008). Recently, it has become clear that GABAergic synaptic plasticity in addition to the widely studied glutamatergic plasticity is targeted by drugs of abuse and drug-induced modulation of this plasticity could critically influence DA neuronal activity and DA release in VTA projection areas, as well as in local VTA microcircuits (Langlois and Nugent, 2017). We previously demonstrated that GABAergic synapses onto VTA DA neurons are able to exhibit a Hebbian heterosynaptic STDP where near-coincident and correlated activities of presynaptic glutamatergic neurons with postsynaptic DA neurons result in expression of LTP and LTD (that we call LTPGABA and LTDGABA). More importantly, we found that GABAergic synapses are predisposed to undergo LTDGABA in response to spike trains (a complex STDP protocol that includes bursts of a combination of pre-post and post-pre spiking; Kodangattil et al., 2013). Somatodendritically released DA within the VTA acts on DA D2-like receptors (D2Rs) and presents an important mechanism for controlling the excitability of DA neurons (Beckstead et al., 2004). There is also an emerging neuromodulatory role for DA in STDP (Pawlak et al., 2010). In fact, our previous study (Dacher and Nugent, 2011) demonstrated that the induction of LTD in response to a traditional LTD pairing protocol (a combination of low frequency stimulation (LFS) with modest depolarization) is dependent on D2R activation. Here we sought to explore the potential effects of pharmacological manipulation of DA transmission within the VTA on the induction of GABAergic STDP in response to complex STDP protocols. We found that DA action through either D1Rs or D2Rs is necessary and sufficient for the induction of LTDGABA in VTA DA neurons and that this DA-dependent plasticity is limited to GABAergic synapses in the VTA. Given that drugs of abuse increase DA neurotransmission within the VTA (Bradberry and Roth, 1989; Klitenick et al., 1992; Campbell et al., 1996; Rahman et al., 2003), our present findings provide an inhibitory synaptic mechanism by which drug-induced alteration of local VTA DA signaling could affect DA cell excitability and subsequently DA release in VTA DA circuits.

## MATERIALS AND METHODS

Brain slice preparation and electrophysiological recordings were conducted as described previously from 14 days to 21 days old Sprague-Dawley rats (Dacher et al., 2013; Kodangattil et al., 2013). Briefly, animals were anesthetized using isoflurane and quickly decapitated. The brain was rapidly dissected and placed into ice-cold artificial cerebrospinal fluid (ACSF) containing (in mM): 126 NaCl, 21.4 NaHCO3, 2.5 KCl, 1.2 NaH2PO4, 2.4 CaCl2, 1.00 MgSO4, 11.1 glucose, 0.4 ascorbic acid, saturated with 95% O2/5% CO2. Horizontal midbrain slices containing the substantia nigra pars compact (SNc) and VTA were cut (250 µm) and incubated in ACSF during at least 1 h at 34◦C. Slices were then transferred into a recording chamber in ascorbic acid-free ACSF at 28◦C. All experiments were carried out in accordance with the National Institutes of Health (NIH) guidelines for the care and use of laboratory animals and were approved by the Uniformed Services University Institutional Animal Care and Use Committee. All efforts were made to minimize animal suffering, and to reduce the number of animals used.

GABA<sup>A</sup> inhibitory post-synaptic currents (IPSCs) were recorded using a patch amplifier (Multiclamp 700B) under infrared-differential interference contrast microscopy. Data acquisition and analysis were performed using DigiData 1440A and pClamp 10 (Molecular Devices, Union City, CA, USA). In all experiments, 6,7-dinitroquinoxaline-2,3-dione (DNQX, 10 µM) and strychnine (1 µM) obtained from Sigma were added to block AMPA- and glycine-mediated synaptic currents, respectively to pharmacologically isolate GABA<sup>A</sup> IPSCs that were completely blocked by the GABA<sup>A</sup> receptor antagonist, bicuculline. Paired GABA<sup>A</sup> IPSCs were evoked using a bipolar stainless steel stimulating electrode placed 200–500 mm rostral to the recording site in the VTA at 0.1 Hz (duration 100 µs, 50 ms inter-stimulation interval) and recorded using KCl containing electrodes and whole-cell voltage-clamp in neurons held at −70 mV. Pipettes were filled with (in mM): 125 KCl, 2.8 NaCl, 2 MgCl2, 2 ATP-Na+, 0.3 GTP-Na+, 0.6 EGTA and 10 HEPES (pH adjusted to 7.28 with KOH, osmolarity adjusted to 275–280 mOsm with sucrose). Stimulation intensity was adjusted to evoke baseline synaptic responses ranged between −200 pA and −800 pA (approximately 50% of maximal responses). The cell input resistance and series resistance were monitored through the experiment and if these values changed by more than 10%, data were not included. The appearance of an Ih current (≥50 pA) in response to stepping cells from −50 mV to

depression (LTDGABA) in response to a complex spiking spike-timing-dependent plasticity (STDP) protocol. Panel (A) represents sample bursts of the complex spiking protocol for induction of LTDGABA. (B,C) Single and average experiments showing induction of STDP recorded in Ih(+) (presumably DA) neurons in VTA (filled square symbols) or SNc (filled circle symbols). At the arrow, STDP was induced. Insets: averaged inhibitory postsynaptic currents (IPSCs) before and 25 min after STDP protocol. In this and all figures, 10 consecutive traces from each condition were averaged for illustration as inset. Calibration: 100 pA, 25 ms (VTA: 75 ± 1.1% of pre-STDP values, F(3,22) = 7.5, p < 0.0001, n = 8; SNc: 103 ± 2.4% of pre-STDP values, F(5,70) = 1.26, n = 14). Values shown throughout figure are the mean ± SEM.

−100 mV was used to identify putative SNc/VTA DA neurons. As a standard protocol in our lab including the present study we consistently record from a region of the VTA (in the dorsal and caudal VTA) that is shown to contain mostly nucleus accumbens (NAc)- projecting DA neurons with Ih positivity (Margolis et al., 2006a,b; Zhang et al., 2010). In addition we consistently consider other electrophysiological criterions for identification of DA neurons (AP characteristics and frequency) that are also commonly used to identify putative DA neurons (Johnson and North, 1992; Dacher et al., 2013; Kodangattil et al., 2013).

To induce STDP, we first obtained a stable baseline for 10 min and then DA cells were taken to current clamp and received trains of a sub-threshold presynaptic stimulation paired with back propagating action potentials (bAPs/postsynaptic spiking) at 5 Hz. To evoke bAPs, cells were injected with direct somatic currents of 1.5 nA for 5 ms through patch pipettes. STDP protocols consisted of 30 trains of five bursts repeated at 0.1 Hz. To induce LTDGABA using post-pre pairing, each burst was composed of three bAPs at 50 Hz followed by a single presynaptic stimulation (negative timing, −5 ms). A complex STDP protocol was used to induce LTDGABA where each burst composed of three bAPs preceded with three presynaptic stimulations at 50 Hz (the complex pairing protocol included both positive timing, +5 ms and negative timing, −15 ms, **Figure 1A**). Values are presented as means ± SEM. Statistical significance was assessed using repeated measures ANOVA with significance level of p < 0.05. Levels of STDP are reported as averaged IPSC amplitudes for 5 min just before STDP induction compared with averaged IPSC amplitudes during the 5 min period from 25 min to 30 min after protocol. Interleaved control experiments were performed with experiments in which drugs were bath applied. Salts and all drugs were purchased from Sigma-Research Biochemicals International or Tocris Bioscience.

# RESULTS

# VTA but Not SNc DA Neurons Exhibit LTDGABA in Response to a Complex Pairing STDP Protocol

Previously, we showed that the synaptic efficacy of GABAergic synapses onto VTA DA neurons can be bi-directionally modified by pre/post spike pairing in a narrow time window (Kodangattil et al., 2013; Authement et al., 2015). Here, we used a more natural and complex pattern of spiking (**Figure 1A**) to induce STDP at GABAergic synapses onto midbrain VTA/SNc DA neurons. Consistent with our previous results (Kodangattil et al., 2013), we were able to induce LTDGABA in VTA DA neurons in response to a combination of both positive (+5 ms) and negative (−15 ms) timing (**Figure 1B**). On the other hand, we found that SNc DA neurons did not exhibit any form of plasticity in response to the same complex STDP protocol (**Figure 1C**). Given this finding, we only examined the effects of D1R/D2R drugs on the induction of STDP (LTDGABA) in VTA DA neurons.

# Induction of LTDGABA in the VTA by STDP Protocols Requires D2R Activation

LTDGABA at GABAergic synapses onto VTA DA neurons can also be triggered in response to a traditional LTD pairing paradigm using LFS paired with modest depolarization (Dacher and Nugent, 2011). LTDGABA triggered in response to both LFS-pairing paradigm and STDP protocols is dependent on the postsynaptic scaffolding A-kinase anchoring protein 79/150 (AKAP79/150) signaling complex which selectively controls GABAergic synaptic strength and mediates the opposing effects of protein kinase A (PKA) and calcineurin (CaN) on GABA<sup>A</sup> receptor trafficking in VTA DA neurons (Dacher and Nugent, 2011; Dacher et al., 2013; Authement et al., 2015). Since LTDGABA in response to the LFS-pairing LTD paradigm is also D2R-dependent and modulated by morphine (Dacher and Nugent, 2011), we tested whether LTDGABA induced by STDP protocols also requires D2R activation. We attempted to induce LTDGABA by a complex STDP protocol or a post-pre STDP protocol as previously described (Kodangattil et al., 2013; Authement et al., 2015) while a D2R antagonist, sulpiride (10 µM), was present in the perfusate throughout the experiment. Sulpiride was able to completely block the induction of LTDGABA in response to the complex STDP protocol (**Figure 2A**) as well as post-pre STDP protocol (**Figure 2B**) suggesting that both the pairing and STDP protocols trigger this D2R-dependent LTDGABA. It should be mentioned that D2R activation by a D2R agonist results in a rundown in GABAergic IPSCs (a chemical form of LTD) in VTA DA neurons that is dependent on CaN activity upon inhibition of PKA-AKAP150 anchoring (Dacher et al., 2013).

# Induction of LTDGABA by a Complex STDP Protocol Also Requires D1R Activation

The roles of D1Rs and D2Rs have been implicated in induction and modulation of STDP (Pawlak and Kerr, 2008; Shen et al., 2008; Pawlak et al., 2010; Xu and Yao, 2010; Ruan

FIGURE 2 | DA D1-like receptor (D1R) and D2R activation is required for induction of LTDGABA in response to the complex STDP protocol. Panels (A–D) show average experiments of STDP with sample traces from Ih(+) neurons in response to the complex or post-pre STDP protocols in drug-free artificial cerebrospinal fluid (ACSF; controls, filled square symbols) or drug bath application (open square symbols) experiments. Control LTDGABA group in response to the complex STDP protocol is similar to Figure 1B in (A,C,D) Representing the interleaved control experiments conducted with drug-treated slice experiments. (A) The D2R antagonist blocked the induction of LTDGABA in response to the complex protocol (sulpiride: 104 ± 1.8% of pre-STDP values, F(4,14) = 0.33, n = 7). (B) Sulpiride prevented the induction of LTDGABA in response to the pre-post protocol while control slices showed robust LTDGABA (control: 69 ± 1.8% of pre-STDP values, F(4,25) = 7.01, p < 0.0001, n = 7; sulpiride: 96 ± 1.5% of pre-STDP values, F(4,32) = 0.63, n = 8). (C) The D1R agonist did not affect the induction of LTDGABA in response to the complex protocol (SKF81297: 75 ± 1.2% of pre-STDP values, F(3,12) = 3.7, p < 0.05, n = 7). (D) The D1R antagonist blocked the induction of LTDGABA in response to the complex protocol (SCH23390: 89 ± 1.04% of pre-STDP values, F(4,32) = 2.01, n = 10). Calibration: 100 pA, 25 ms.

et al., 2014). Although DA via both D1R and D2R may exert opposing effects on STDP where D1Rs enable the induction of LTP and D2Rs mainly favor LTD, the cooperative actions of both DA receptors in induction of LTD of STDP have been shown (Xu and Yao, 2010). Here we tested our hypothesis that D1R activation blocks LTDGABA and reverses the direction of plasticity to promote LTPGABA. To test this, we attempted to induce LTDGABA in response to the complex pairing STDP protocol in the presence of a D1R agonist (SKF81297, 10 µM) or a D1R antagonist (SCH233390, 10 µM). While LTDGABA was unaffected in the presence of the D1R agonist (**Figure 2C**), the D1R antagonist blocked the induction of LTDGABA (**Figure 2D**) suggesting that D1R activation also facilitates the induction of LTD at GABAergic synapses onto VTA DA neurons.

### DISCUSSION

Here, we have extended our previous studies on GABAergic STDP in VTA DA neurons to investigate how local VTA DA signaling through somatodendritic release of DA during STDP within the VTA would affect STDP. We found that while GABAergic synapses onto VTA DA neurons were able to express LTDGABA in response to a complex STDP protocol, GABAergic synapses onto SNc DA neurons did not show any form of plasticity in response to the same protocol. This suggests that this form of inhibitory plasticity may be selectively expressed at GABA<sup>A</sup> synapses in the VTA. LTDGABA triggered by our complex STDP protocol or a post-pre STDP protocol in VTA DA neurons was dependent on D2R activation similar to LTDGABA triggered in response to a typical LFS-pairing protocol shown by our group (Dacher and Nugent, 2011). D1R activation seems to commonly facilitate the induction of LTP in response to STDP protocols although it could also contribute to induction of LTD of STDP at the synapse (Pawlak and Kerr, 2008; Zhang et al., 2009; Ruan et al., 2014; Brzosko et al., 2015). Thus, we further examined whether boosting endogenous DA action on D1R activity during STDP induction through bath application of a D1R agonist could reverse the direction of plasticity towards LTP. Not only were we unable to trigger LTP in response to the complex protocol in the presence of D1R agonist, the induction of LTDGABA was completely unaffected ruling out the possibility of masking LTD by a simultaneous induction of a D1R-dependent LTP at these synapses. Interestingly, we observed a blockade of LTDGABA in response to the complex protocol in the presence of D1R antagonist suggesting that local endogenous DA could engage D1Rs to promote this plasticity. The selectivity of induction of GABAergic STDP at GABAergic synapses in the VTA vs. the SNc may not be surprising considering the distinct anatomical and functional populations of VTA and SNc DA neurons (Beier et al., 2015; Shin et al., 2017). Given that VTA DA neurons are found to be more heterogeneous than SNc DA neurons (Margolis et al., 2006b) and the postsynaptic nature of DA-dependent LTDGABA triggered at GABAergic synapses in the VTA (Dacher and Nugent, 2011; Dacher et al., 2013; Kodangattil et al., 2013; Authement et al., 2015), our data suggest that the expression of this STDP as a uniform property of GABA<sup>A</sup> synapses in the VTA may be due to distinct intrinsic characteristics of the VTA microcircuits that differ from the SNc local circuitry. Consistently, it has been shown that extracellular DA levels released within the VTA and SNc differ as the dendritic release of DA within the VTA is far greater than DA release within the SNc (Rice et al., 1997; Ford et al., 2010). Therefore, we propose that this regional difference in release properties of DA at dendritic locations may underlie the selective expression of STDP in the VTA. Given that activation of either D1Rs or D2Rs by endogenous DA was sufficient to trigger this plasticity and considering different localization of DA receptors in VTA neuronal populations and presynaptic terminals innervating VTA neurons, it remains to be known how and where the activation of either presynaptic or postsynaptic DA receptors mediate this plasticity in the VTA. GABA<sup>A</sup> synapses onto VTA DA neurons mainly originate from VTA GABAergic neurons (that comprise 30% of VTA neuronal populations) and rostromedial tegmental area (RMTg) neurons (Barrot et al., 2012). D2Rs are mainly expressed postsynaptically on VTA DA neurons to provide an auto feedback inhibition of DA neurons (Beckstead et al., 2007). D2Rs are assumed to be expressed on GABAergic terminals where their activation facilitates the induction of a presynaptic endocannabinoidmediated LTD at GABAergic synapses onto VTA DA neurons (Pan et al., 2008). We have also shown that inhibition of PKA activity (which is the main downstream signaling mechanism for both D1R and D2R) or disruption of AKAP150-PKA association promotes LTDGABA in response to STDP protocols by favoring CaN activity and endocytosis of GABA<sup>A</sup> receptors in VTA DA neurons (Authement et al., 2015). Given that D2Rs can inhibit PKA activity and the postsynaptic locus of LTDGABA expression, we assume that D2R activation acts through this signaling pathway to promote LTDGABA in response to complex STDP protocols. Only a small subset of VTA DA neurons express D1Rs (D1/D5; Schilström et al., 2006). DA increases presynaptic release of GABA in the midbrain through D1R activation (Cameron and Williams, 1993). It has been shown that NAc D1R-expressing medium spiny neuronal projections inhibiting VTA DA neurons preferentially make GABA<sup>B</sup> synapses onto VTA DA neurons while these D1R expressing neurons of NAc inhibit VTA GABAergic interneurons via activating GABA<sup>A</sup> synapses (Barrot et al., 2012). It will be interesting to test whether the postsynaptic action of DA on D1Rs in VTA DA neurons promotes LTD through modulation of NMDA receptor (Schilström et al., 2006) or GABA<sup>B</sup> receptor activity originating from D1 expressing NAc (Cameron and Williams, 1993; Kamikubo et al., 2007) in VTA DA neurons. It should be noted that there are several limitations to our study including the young age of the rats, the suboptimal cooler temperature for recordings at 28◦C, the use of electrical stimulation rather than optogenetic stimulation of specific afferent inputs to the VTA, and also the complexity of mimicking a natural pattern of neuronal firings in an in vitro STDP induction protocol. In fact, a recent study has shown the heterogeneity of distinct GABAergic inputs to VTA DA neurons where only GABAergic inputs arising from the VTA GABAergic interneurons show short-term plasticity (Polter et al., 2018). In sum, we demonstrated that synaptic actions of DA within the VTA is required for induction of GABAergic STDP. The selective expression of this DA-dependent STDP in the VTA presents an important synaptic learning mechanism that can be targeted by drugs of abuse or stress to alter DA signaling within VTA DA circuits and significantly impact reward-related behavioral learning.

#### DATA AVAILABILITY

All relevant data is contained within the manuscript: all datasets (GENERATED/ANALYZED) for this study are included in the manuscript.

#### AUTHOR CONTRIBUTIONS

FN designed the experiments. LL, MD and FN performed electrophysiology experiments, analyzed the data and prepared the figures. LL and FN wrote the manuscript.

#### REFERENCES


#### FUNDING

This work was supported by the National Institutes of Health (NIH)—National Institute on Drug Abuse (NIDA) Grant#R01 DA039533 to FN. The funding agency did not contribute to writing this article or decide in submitting the article.

#### ACKNOWLEDGMENTS

The opinions and assertions contained herein are the private opinions of the authors and are not to be construed as official or reflecting the views of the Uniformed Services University of the Health Sciences or the Department of Defense or the Government of the United States. We are grateful to Nugent Laboratory members, Ryan Shepard and Robert Hammack for their constructive comments on the manuscript.


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Langlois, Dacher and Nugent. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Data Driven Models of Short-Term Synaptic Plasticity

Elham Bayat Mokhtari <sup>1</sup> , J. Josh Lawrence<sup>2</sup> and Emily F. Stone<sup>1</sup> \*

<sup>1</sup> Department of Mathematical Sciences, The University of Montana, Missoula, MT, United States, <sup>2</sup> Pharmacology and Neuroscience, Texas Tech University Health Sciences Center, Lubbock, TX, United States

Simple models of short term synaptic plasticity that incorporate facilitation and/or depression have been created in abundance for different synapse types and circumstances. The analysis of these models has included computing mutual information between a stochastic input spike train and some sort of representation of the postsynaptic response. While this approach has proven useful in many contexts, for the purpose of determining the type of process underlying a stochastic output train, it ignores the ordering of the responses, leaving an important characterizing feature on the table. In this paper we use a broader class of information measures on output only, and specifically construct hidden Markov models (HMMs) (known as epsilon machines or causal state models) to differentiate between synapse type, and classify the complexity of the process. We find that the machines allow us to differentiate between processes in a way not possible by considering distributions alone. We are also able to understand these differences in terms of the dynamics of the model used to create the output response, bringing the analysis full circle. Hence this technique provides a complimentary description of the synaptic filtering process, and potentially expands the interpretation of future experimental results.

#### Edited by:

Vito Di Maio, Istituto di Scienze Applicate e Sistemi Intelligenti "Eduardo Caianiello" (ISASI), Italy

#### Reviewed by:

Gianluigi Mongillo, Université Paris Descartes, France Andrey Olypher, Georgia Gwinnett College, United States Katsunori Kitano, Ritsumeikan University, Japan

#### \*Correspondence:

Emily F. Stone stone@mso.umt.edu

Received: 01 February 2018 Accepted: 27 April 2018 Published: 22 May 2018

#### Citation:

Bayat Mokhtari E, Lawrence JJ and Stone EF (2018) Data Driven Models of Short-Term Synaptic Plasticity. Front. Comput. Neurosci. 12:32. doi: 10.3389/fncom.2018.00032 Keywords: short term plasticity, epsilon machines, synaptic filtering, mutual information, interneuron-pyramidal cell synapses, causal state splitting reconstruction

# 1. INTRODUCTION

Short term plasticity at the synapse level can have profound effects on functional connectivity of neurons. Through repetitive activation, the strength, or efficacy, of synaptic release of neurotransmitter can be decreased, through depletion, or increased, through facilitation. A single synapse type can display different properties at different frequencies of stimulation.

The role of synaptic plasticity and computation has been analyzed and reported on in numerous papers over the past 30 years. A review of feed-forward synaptic mechanisms and their implications can be found in Abbott and Regehr (2004). In this paper Abbott and Regher state "The potential computational power of synapses is large because their basic signal transmission properties can be affected by the history of presynaptic and postsynaptic firing in so many different ways." They also outline the basic function of a synapse as a signal filter as follows: Synapses with an initial low probability of release act as high pass filters through facilitation, while synapses with an initially high probability of release exhibit depression and subsequently serve as low pass filters. Intermediate cases in which the synapse can act as a band-pass filter, exist. Identifying synapse-specific molecular mechanisms is currently an active area of research, involving subtle changes in expression of myriad calcium sensor isoforms (synaptotagmins), subtly configured to alter the microscopic rates of synaptic release, facilitation, depression, and vesicle replenishment (Fioravante and Regehr, 2011; Chen and Jonas, 2017; Jackman and Regehr, 2017).

The underlying mechanisms creating these effects may be inferred by fitting an a priori model to synaptic response data. We parameterize such a model combining the properties of facilitation and depression (FD) at the presynaptic neuron with experimental data from dual whole-cell recordings from a presynaptic parvalbumin-positive (PV) basket cell (BC) connected to a postsynaptic CA1 (Cornu Ammonis 1 subregion) pyramidal cell, for fixed frequency spike trains into the presynaptic PV BC (Stone et al., 2014; Lawrence et al., 2015). We later examine the response of the model to an in vivo-like Poisson spike train of input, where the inter-spike interval (ISI) follows an exponential distribution, in Bayat et al. (submitted). Here we investigate the information processing properties of the synapse in question, following (Markram et al., 1998) and using standard calculations of entropy and mutual information between the input spike train and output response. This, however, left us unsatisfied, as it did not indicate the history dependence of the response, which we believe is one of the more interesting features of plasticity models that involve presynaptic calcium concentration. We attempted using multivariate mutual information measures, but this very quickly collapses due to the "curse of dimensionality" (Bellman, 1957). In this paper we try to resolve the question using methods of Computational Mechanics, creating unifilar HMMs called epsilon machines, that represent the stochastic process that our synapse model creates. As a bonus we are using data itself (albeit synthetic data) to create models of plasticity that can be used to classify properties of different types of synapses.

As stated in the abstract, methods from Information theory rely on distribution measures which inherently ignore the ordering of the measured data stream. We seek to incorporate this important feature of plasticity, the dependence of the response of the synapse on the prior sequence of stimulation, directly through the construction of causal state machines. This can only add to the understanding of the process in cases where the input stimulus train is known. In experiments where only the output postsynaptic response is known, this technique is particularly useful. While the machines themselves cannot be interpreted in a physiological way, the information they provide can be used to classify synaptic dynamics and inform the construction of physiological models. The point of the analysis is to gain as much accurate information from experiments in short term synaptic plasticity as possible without imposing the bias of an assumed underlying physical model. To create synthetic data we use a very simple but otherwise complete model of short-term plasticity that incorporates a "memory" effect through the inclusion of calcium build-up and decay. This has roots in a real physiological process (the flooding of calcium into the presynaptic terminal can trigger the release of neurotransmitter), but we are not interested per se in creating a biophysically complete model here. The calcium dynamics simply introduces another time scale into the model, one that is physiologically relevant. We wish to explore the effect of this additional time scale on the complexity of the process.

Computational Mechanics is an area of study pioneered by Crutchfield and colleagues in the 1990s, (Crutchfield and Young, 1989; Crutchfield, 1994; Shalizi and Shalizi, 2002). Finding structure in time series with these techniques has been applied in such diverse arenas as layered solids (Varn et al., 2002), Geomagnetism (Clarke et al., 2003), climate modeling (Palmer et al., 2002), financial time series (Park et al., 2007), and more recently ecological models (Boschetti, 2008) and large scale multi-agent simulations (Parikh et al., 2016). In neuroscience, to name a few only, we note one application to spike train data (Haslinger et al., 2013), and a recent publication by Marzen et al. on the time resolution dependence of information measures of spike train data (Marzen et al., 2015).

We employ some of the simplest ideas from this body of work, namely decomposing a discrete time-discrete state data stream into causal states, which are made up of sequences of varying length that all give the same probability of predicting the same next data point in the stream. The data are discrete time by construction, and made into discrete symbols through a partition, so the process can be described by symbolic dynamics. We use the Causal State Splitting Reconstruction (CSSR) algorithm on the data to create the causal states and assemble a HMM that represents the transitions between the states, while emitting a certain symbol. This allows us to classify the synapse types and gives an idea of the differences in the history dependence of the processes as well.

Using an a priori model for short-term synaptic dynamics and fitting it to data, while a perfectly valid approach, allows only for the discovery of the parameters in the model and possibly a necessary model reduction to remove any parameter dependencies (too many parameters in the model for the data set to fit). The alternative approach is to allow the data itself to create the model. From these "data driven" models, conclusions can be drawn about the properties of the synapse that are explicitly discoverable from the experimental data. The ultimate goal is a categorization of the types of processes a synapse can create, and an assignment of those to different synapse types under varying conditions. Note that the complexity or level of biophysical detail of our model synapse is not important to this end. In fact, the best way to calibrate this method is using the simplest possible model of the dynamics that captures the history dependence of the plasticity. This is not consonant with the goal of incorporating as many physiological features as possible, whether they affect the dynamics significantly or not. In fact, in most cases the limited data in any electro-physiological experiment precludes identifying more than a handful of parameters in an a priori model, a point we discuss in Stone et al. (2014). Our goal is to classify the sort of filter the synapse creates under certain physiological conditions, rather than to identify specific detailed cellular level mechanisms.

We are motivated in this task by the work of Kohus et al. (2016), in which they present a comprehensive data set describing connectivity and synaptic dynamics of different

**Abbreviations:** BC, basket cell; CA1, Cornu Ammonis, early name for hippocampus; FD, facilitation and depression; IPSC, inhibitory postsynaptic current; ISI, interspike interval; KL, Kozachenko and Leonenko; KSG, Kraskov, Stögbauer, and Grassberger; mAChR, muscarinic acetylcholine receptors; MCMC, Monte Carlo Markov Chain; MI, Mutual Information; NT, neurotransmitter; PSR, postsynaptic response; PV, parvalbumin-positive; HMM, Hidden Markov Model; CSSR, Causal State Splitting Reconstruction.

interneuron (IN) subtypes in CA3 using paired cell recordings. They apply dynamic stimulation protocols to characterize the short-term synaptic plasticity of each synaptic connection across a wide range of presynaptic action potential frequencies. They discovered that while PV+ (parvalbumin positive) cells are depressing, CCK+ (Cholecystokinin positive) INs display a range of synaptic responses (facilitation, depression, mixed) depending upon postsynaptic target and firing rate. Classifying such a wide range of activity succinctly is clearly useful in this context. The discovery that the rate of particular observed oscillations in these cells (called sharp wave ripples) may be paced by the shortterm synaptic dynamics of the PV+BC in CA3 demonstrates the importance of these dynamics in explaining complex network phenomena.

The paper is organized as follows. The construct for an experimental paper with section 2 and section 3 is not an immediately obvious partition of our work, but we use it as best we can. In the section 2 we describe the background on the synaptic plasticity model, and some analysis of its properties. We also cover the necessary background from Computational Mechanics. Finally we show how the techniques are explicitly applied to our data. In the section 3 we present the epsilon machines created from data from three types of synapses from our FD model: depressing, facilitating, and mixed, at varying input frequencies. Here we also indicate similarities and differences in the actual machines. In the section 4 we speculate on the reasons for these features by referring back to the original model. In the last section we indicate directions for future work.

### 2. MATERIALS AND METHODS

#### 2.1. Model of Synaptic Plasticity

In Stone et al. (2014), we parameterize a simple model of presynaptic plasticity from work by Lee et al. (2008) with experimental data from cholinergic neuromodulation of GABAergic transmission in the hippocampus. The model is based upon calcium dependent enhancement of probability of release and recovery of signalling resources (For a review of these mechanisms see Khanin et al., 2006). It is one of a long sequence of models developed from 1998 to the present, with notable contributions by Markram et al. (1998) and Dittman et al. (2000). The latter is a good exposition of the model as it pertains to various types of short term plasticity seen in the central nervous system, and the underlying dependence of the plasticity is based on physiologically relevant dynamics of calcium influx and decay within the presynaptic terminal. In our work, we use the Lee model to create a two dimensional discrete dynamical system in variables for calcium concentration in the presynaptic area and the fraction of sites that are ready to release neurotransmitter into the synaptic cleft.

In the rest of this section we outline the model, which is used to generate synthetic data for our study of causal state models, or epsilon machines, of short-term plasticity.

In the model the probability of release (Prel) is the fraction of a pool of synapses that will release a vesicle upon the arrival of an action potential at the terminal. Following the work of Lee et al. (2008), we postulate that Prel increases monotonically as function of calcium concentration in a sigmoidal fashion to asymptote at some Pmax. The kinetics of the synaptotagmin-1 receptors that binds the incoming calcium suggests a Hill equation with coefficient 4 for this function. The half-height concentration value, K, and Pmax are parameters determined from the data.

After releasing vesicles upon stimulation, some portion of the pool of synapses will not be able to release vesicles again if stimulated within some time interval, i.e., they are in a refractory state. This causes "depression;" a monotonic decay of the amplitude of the response upon repeated stimulation. The rate of recovery from the refractory state is thought to depend on the calcium concentration in the presynaptic terminal (Dittman and Regehr, 1998; Wang and Kaczmarek, 1998). The model has a simple monotonic dependence of rate of recovery on calcium concentration, a Hill equation with coefficient of 1, starting at some kmin, increasing to kmax asymptotically as the concentration increases, with a half height of K<sup>r</sup> .

The presynaptic calcium concentration itself, [Ca], is assumed to follow first order decay kinetics to a base concentration, [Ca]base. At this point we choose that [Ca]base = 0, since locally (near the synaptotagmin-1 receptors) the concentration of calcium will be quite low in the absence of an action potential. The evolution equation for [Ca] then is simply τca d[Ca] dt = −[Ca] where τca is the calcium decay time constant, measured in milliseconds. Upon pulse stimulation, presynaptic voltage-gated calcium channels open, and the concentration of calcium at the terminal increases rapidly by an amount δ (measured in µm): [Ca] → [Ca] + δ at the time of the pulse. Note that calcium build-up is possible over a train of pulses if the decay time is long enough relative to the ISI.

As mentioned above, the probability of release Prel and the rate of recovery, krecov, depend monotonically on the instantaneous calcium concentration via Hill equations with coefficients of 1 and 4 respectively. Rescaling the calcium concentration by δ = δ<sup>c</sup> in the control case, we define C = [Ca]/δ<sup>c</sup> . Then the equations are Prel = Pmax C 4 <sup>C</sup>4+K<sup>4</sup> , and krecov = kmin + 1k C C+K<sup>r</sup> . The variable Rrel is governed by the ordinary differential equation dRrel dt = krecov(1 − Rrel), which can be solved exactly for Rrel(t). <sup>R</sup>rel(t) <sup>=</sup> <sup>1</sup> <sup>−</sup> (1 <sup>−</sup> <sup>R</sup>0)( <sup>C</sup>0<sup>e</sup> <sup>−</sup>t+K<sup>r</sup> Kr+C<sup>0</sup> ) 1k e −kmint . Prel is also a function of time as it follows the concentration of calcium after a stimulus.

We used experiments in hippocampus to parameterize this model, as part of an exploration of the frequency dependent effects of neuromodulation. Whole-cell recordings were performed from synaptically connected pairs of neurons in mouse hippocampal slices from PV-GFP mice (Lawrence et al., 2015). The presynaptic neuron was a PV basket cell (BC) and the postsynaptic neuron was a CA1 pyramidal cell. Using short, 1–2 ms duration suprathreshold current steps to evoke action potentials in the PV BC from a resting potential of −60 mV and trains of 25 of action potentials are evoked at 5, 50, and 100 Hz from the presynaptic basket cell. The result in the postsynaptic neuron is the activation of GABAA-mediated inhibitory postsynaptic currents (IPSCs). Upon repetitive stimulation, the amplitude of the synaptically evoked IPSC declines to a steady-state level. These experiments were conducted with 5, 50, and 100 Hz stimulation pulse trains, with and without the neuromodulator muscarine, in order to test frequency dependent short term plasticity effects.

The peak of the measured postsynaptic IPSC is presumed to be proportional to the total number of synapses that receive stimulation Ntot, which are also ready to release (Rrel), e.g., NtotRrel, multiplied by the probability of release Prel. That is, peak IPSC ∼ NtotRrelPrel. Prel and Rrel are both fractions of the total, and thus range between 0 and 1. Without loss of generality, we consider peak IPSC to be proportional to RrelPrel.

From the continuous time functions describing C, Rrel, and Prel, we constructed a discrete dynamical system (or "map") that describes PrelRrel upon repetitive stimulation. Given an ISI of T, the calcium concentration after a stimulus is C(T) + 1 (1 = δ/δc), and the peak IPSC is proportional to Prel(T)Rrel(T), which depend upon C. After the release, Rrel is reduced by the fraction of synapses that fired, e.g., Rrel → Rrel−PrelRrel = Rrel(1−Prel). This value is used as the initial condition in the solution to the ODE for Rrel(t). A two dimensional map (in C and Rrel) from one peak value to the next is thus constructed. To simplify the formulas we let P = Prel and R = Rrel. The map is

$$\mathcal{C}\_{n+1} = \mathcal{C}\_n e^{-T} + \Delta,\tag{1}$$

$$P\_{n+1} = P\_{\max} \frac{C\_{n+1}^4}{C\_{n+1}^4 + K^4},\tag{2}$$

$$R\_{n+1} = 1 - (1 - (1 - P\_{\text{fl}})R\_{\text{fl}}) (\frac{C\_n e^{-T} + K\_r}{K\_r + C\_n})^{\Delta k} e^{-k\_{\text{min}}T} . \tag{3}$$

Following this notation the peak value upon the nth stimulus is Pr<sup>n</sup> = RnPn, where R<sup>n</sup> is the value of the reserve pool before the release reduces it by the fraction (1 − Pn).

Data from the experiments were fitted to the map using the Matlab package lsqnonlin, and with Bayesian techniques (Haario et al., 2006). The value of Pmax was determined by variancemean analysis, and is 0.85 for the control data and 0.27 for the muscarine data. The common fitted parameter values for both data sets are shown in **Table 1**.

For the control data set 1 = 1, and the muscarine data set has the fitted value of 1 = 0.17. From this result it is clear that the size of the spike in calcium during a stimulation event must be much reduced to fit the data from the muscarine experiments. This is in accordance with the idea that mAChR activation reduces calcium ion influx at the terminal.

#### 2.1.1. Analyzing the Map

It is common in the experimental literature to classify a synapse as being "depressing" or "facilitating," depending upon its response to a pulse train at some relevant frequency. Simple models can be built that create each effect individually. The model here combines both mechanisms so that, depending upon the parameters, both facilitation and depression and a mixture of the two can be represented (Lee et al., 2008). Note that facilitation is built into this model through the calcium dependent P value and rate of recovery. For instance, by varying the parameters we can create a "mock" facilitating synapse, where the size of the response increases with increasing frequency of input stimulation, or a "mixed" synapse, where the response is depressed for low and high frequency, but increases comparatively for moderate values of the frequency.

We are able to "tune" the parameters in the map from the fitted values to realize these cases, and the results are shown in **Table 2**. To attain more complicated dynamics we must first increase the calcium decay time to 30 ms, a much larger value that has neverthe-less been found in fitting the model to electrophysiological data from other synapses (Lawrence lab, unpublished results). The build-up of calcium means a larger recovery rate, but the probability of release ranges only up to Pmax = 0.6, and over a larger concentration range of the calcium (K = 4.0 for facilitating and K = 1.0 for mixed), off-setting the effect of the larger amount of calcium from the build-up to a varying degree. The competition between increasing probability of release and decreasing R creates the local maximum in the mixed case, which is also present in the facilitating case, but for frequencies outside the physiological range.

The map has a single attracting fixed point, and the collapse to this fixed point from physiological initial conditions is very rapid (Stone et al., 2014). The value of the fixed point depends on the frequency (1/T), and plotting this is a good way to represent the different types of synaptic dynamics. In **Figure 1** we plot the expression for the fixed point (Pr = P × R or PR) of the deterministic map vs. rate for three cases. For instance, the depressing synapse fixed point decreases from Pmax (for one stimulus, or zero frequency) monotonically, with a quick decay over 0–10 Hz, and a slower decay to zero following. The facilitating synapse fixed point increases over the physiological range shown, but decreases for larger values of the frequency. The mixed synapse fixed point starts at a base value of 0.3 for one stimulus, increases to a local max near 50 Hz and decays thereafter. The "resonance" indicated by




the local maximum gives the mixed synapse more complicated linear filtering properties than the other two in the physiological frequency range.

#### 2.1.2. The Depressing Synapse

The interplay of the presynaptic probability of release and the rate of the recovery can create a non-linear filter of an incoming stimulus train. To investigate this idea, in Bayat et al. we consider the distribution of values of Pr created by exponentially distributed random ISIs for varying rates λ, or mean ISI, denoted < T >= 1/λ for the depressing synapse. Doing so explores the filtering properties of the synapse when presented with a Poisson spike train. We also present results from numerical studies to determine of the effect of varying the mean rate of the pulse train. The information processing properties, in the form of mutual information and multivariate mutual information, of the synapse at physiological frequencies are compared. We found that the mutual information peaked around 3 Hz, when the entropy of the Pr distribution was at its maximum, for both muscarine and control parameter sets.

We also determined that the random variable describing the calcium concentration has a closed form distribution, and indeed a well-known distribution. However, this is not the case for the variable R due to the complexity of the map, and so a closed form for the distribution of Pr = PR is not possible. However, we can understand it partially by considering the mechanisms involved, and using some information from the deterministic map, namely the expression for the fixed point. If the Pr value is directly determined by the fixed point value for the ISI preceding it, we would be able to convert the distribution of the ISIs into that of the Prs by using composition rules for distributions of random variables. We examine this when the calcium decay time (τca) is notably smaller than the ISI (T). With this approximation C, P, and R have time in between pulses to decay to their steady state value before another pulse. This means that the fixed point value for a rate given by 1/T where T is the preceding interspike interval is more likely to give a good estimate of the actual value or Pr = PR.

It was shown in Stone et al. (2014) that in this case C → 1 as T increases and hence P → Pmax. Therefore, the fixed point for R, (R) is then

$$\overline{R} = \frac{1 - e^{-k\_{\min}T}}{1 - (1 - P\_{\max})e^{-k\_{\min}T}}.$$

With this simplification we found the probability density function (PDF) of R given an exponential distribution of the variable T. For simplicity of notation, we use X = R and Y = PR.

If X is a random variable, then an analytic expression for its PDF is given by

$$f\left(\mathbf{x}|\lambda,\mathbf{c},k\_{\min}\right) = \frac{\lambda(1-\mathbf{c})}{k\_{\min}}(1-\mathbf{x})^{-(1-\lambda/k\_{\min})}(1-\mathbf{c}\mathbf{x})^{-(1+\lambda/k\_{\min})}\,\mathrm{s},\tag{4}$$

where c = 1 − Pmax, λ > 0 is the mean Poisson rate and kmin > 0 is the baseline recovery rate. The distribution is supported on the interval [0, 1]. Similarly, we can compute the analytical expression of the PDF of fixed point Y. We will refer to this in what follows as the stochastic fixed point. Hence, the PDF for the stochastic fixed point is

$$\begin{cases} f\left(\boldsymbol{\wp}|\lambda,\varepsilon,k\_{\min}\right) = \\ \frac{\lambda P\_{\max}(1-\varepsilon)}{k\_{\min}} (P\_{\max}-\boldsymbol{\wp})^{-(1-\lambda/k\_{\min})} (P\_{\max}-\boldsymbol{\wp})^{-(1+\lambda/k\_{\min})}. \end{cases} \tag{5}$$

This distribution is supported on the interval [0, Pmax]. **Figure 2** shows this expression for different mean input ISI, in milliseconds.

In **Figure 3** are histograms of Pr-values obtained numerically from the map with very small τca, with an exponentially distributed T random variable and other parameters from the control set, as in **Figure 2**. The similarity between the two is evident. Apparently this approximation captures the shape of the distribution and how it changes with varying input spike train rate.

We are now convinced that we understand the primary driver of the variation of the probability distribution of the response to the input mean rate. However, as mentioned before, the creation of a distribution automatically ignores the causality in the sequence of the responses. In the next section we describe a method for assessing this causality directly from the response data.

#### 2.2. Computational Mechanics Background

We can use distribution to compute measures of information transfer between input spike trains and output Prs. However, the question of how far back in a spike train the synapse "remembers," or, how far back in the spike train is important for predicting the output, is difficult to answer, even using multivariate mutual information measures. Instead we propose a method for describing the process in terms of output only, with the goal of classifying the complexity of the underlying synaptic

dynamics. This method relies on the ideas of "computational mechanics" developed by Crutchfield and colleagues in the 1990s.

The material presented in this section is drawn from many of the seminal papers by Crutchfield et al. (Crutchfield and Young, 1989; Crutchfield, 1994; Shalizi and Shalizi, 2002), much of which is quite technical. In what follows we outline the key ideas that we have used in our analysis, but note that the theoretical underpinnings of the ideas are completely described in this body of work and we refer the reader to these papers for more detail.

Imagine a black box experimental system and its measurement channel. Inside the black box is a three state system or process. The measurements are a sequence of symbols (0, 1) generated upon transitions between the unseen states in the black box. The measurement channel itself acts to map the internal state sequence · · · BCBAA · · · to a measurement sequence of symbols · · · 01110 · · · . The black box system is assumed to be Markovian, meaning that the transition probability from one state to another depends only upon the current state. The observed symbol sequence, generated upon transitions between states, make the system a hidden Markov process. From the point of view of the observer, how many of the system's properties be inferred from the observed symbol sequence? Can a model of the hidden process be created from this data stream? Can the model be used to predict the future symbols in the sequence?

Let the symbol sequence be represented by S. With information from the past ←−S , we want to make a prediction about the future −→S . The formative idea is to find past sequences of measurements (histories) leading to the same future. Once these states are identified, the transitions between them can be inferred from S. The states themselves and the transition matrix are called the ǫ-machine for the process. A finite state ǫ-machine is a Unifilar Hidden Markov Model given by <sup>M</sup> = {S,{<sup>T</sup> (s) ,s ∈ <sup>A</sup>}} where unifilar means for each state <sup>σ</sup><sup>i</sup> <sup>∈</sup> <sup>S</sup> and each symbol <sup>x</sup> <sup>∈</sup> <sup>X</sup> there is at most one outgoing edge from state <sup>σ</sup><sup>i</sup> and output symbol x.

An ǫ-machine captures the (temporal) patterns in the observations and reflects the causal structure of the process. With this model, we can extrapolate beyond the original observations to predict future behavior of a system. The ǫ-machine is further defined to be the unique, minimal and maximally optimal model of the observed process. It can model stationary stochastic processes with states that represent equivalence classes of histories with no significant difference in their probability distribution over the future events.

#### 2.2.1. Epsilon Machine Construction

Consider a portion of a contiguous chain of random variables: X<sup>n</sup> : <sup>m</sup> = XnXn+<sup>1</sup> · · · Xm, m > n. A semi-infinite chain is either: X<sup>n</sup> : = XnXn+<sup>1</sup> · · · , which is called the future, or X: <sup>n</sup> = · · · Xn−2Xn−1, the past. The bi-infinite chain of random variables is denoted X: . A process is specified by the distribution Prob(X:).

Assume the process is stationary and that a realization of length L has this property: Prob(X<sup>1</sup> : <sup>L</sup>) = Prob(X<sup>n</sup> : <sup>L</sup>+n−1) for all n ∈ **Z**. The values of X<sup>i</sup> , the x<sup>i</sup> , are drawn from a finite alphabet, A. In our case we use two symbols, 0 and 1, and a sample finite realization of the process would look like: 00111001001, for instance.

A causal state σ <sup>+</sup> <sup>∈</sup> <sup>S</sup> + is a set of pasts grouped by the equivalence relation ∼+:

$$\left(\mathbb{x}\_{\colon 0} \curvearrowright^{+} \mathbb{x}\_{\colon 0}^{\prime} \curvearrowright \simeq \succeq \text{Prob}(X\_0 | X\_{\colon 0} = \mathbb{x}\_{\colon 0}) = \text{Prob}(X\_0 | X\_{\colon 0} = \mathbb{x}\_{\colon 0}^{\prime})$$

Two histories are equivalent if and only if they have the same conditional distribution of futures. Groups of specific blocks, e.g., 011, 10, 1011 might all be in the same causal state. At a time t, S + t is a random variable drawn from σ <sup>+</sup> <sup>∈</sup> <sup>S</sup> + and · · · S + −1 S + 0 S + 1 · · · S + t is a causal state process. Each causal state has a future morph Prob(X<sup>t</sup> :|σ + t ), the conditional measure of futures that can be generated from it. Each state inherits a probability π(σ + t ) from the processes measure over all pasts Prob(X: <sup>t</sup>). A generative model is constructed out of the causal states by giving the causal state process transitions:

$$T\_{\sigma\sigma'}^{(\mathbf{x})} = \operatorname{Prob}(\mathbb{S}\_{t+1}^+ = \sigma', X\_t = \mathbf{x} | \mathbb{S}\_t^+ = \sigma)$$

that give the probability of generating the next symbol x and while starting from state σ and ending in state σ ′ . A process' forward-time <sup>ǫ</sup>-machine is the tuple {A, <sup>S</sup> <sup>+</sup>,{T (x) : <sup>x</sup> <sup>∈</sup> <sup>A</sup>}} For a discrete time, discrete alphabet process, the ǫ machine is its minimal unifilar HMM. Minimal means the smallest number of states, and unifilarity means the next state is known given the current state and the next symbol. E.g., the probability of the transition Prob(S + t+1 <sup>|</sup>X<sup>t</sup> <sup>=</sup> <sup>x</sup>, <sup>S</sup> + <sup>t</sup> = σ) has support on a single causal state. The statistical complexity of an epsilon machine is defined to be the entropy of the causal state distribution, e.g., H[S +].

The task of creating the epsilon machine is not a simple one and generally is quite computationally intense. There has been much work on creating code for this purpose, and we rely on available software. For instance, in these preliminary results we use the Causal-State Splitting Reconstruction Algorithm (CSSR) (Shalizi and Klinkner, 2004) to create the machine from blocks of length L starting with L = 1 and increasing up to an appropriate maximum. We note that there packages created by Crutchfield's group that use a Bayesian approach for finding machines, resulting in distributions of possible machines on the level transition probabilities for fixed model topology or for inferring the model topology itself (Travers and Crutchfield, 2011; Strelioff and Crutchfield, 2014).

#### 2.3. Distributions

To create approximations to the distribution of Pr-values we computed 2<sup>15</sup> samples from the stochastic map, after discarding a brief initial transient. The values, ranging between 0 and 1, were placed into evenly spaced bins. The histograms, normalized to be frequency distributions, were computed for a range of mean frequencies (or rates) in the theta range, gamma range, and higher (non-physiological, for comparison). We tested the three different synapse types: depressing, facilitating, and mixed. For parameter values of each, see **Tables 1, 2**. The histograms themselves are shown in **Figures 4**–**6**.

In order to create epsilon machines, the Pr-values must be partitioned into a sequence of 0's and 1's, which requires the

adoption of a threshold value. The choice of this threshold impacts the result, as might be expected. We explore this dependence in Appendix 2, where we show that most of the machines are robust within a finite interval around the chosen threshold. This partition of the output of a real valued map on the interval [0, 1] into a discrete symbol sequence is known as a "symbolic dynamic" and has been studied extensively in dynamical systems theory. For an introductory reference to the mathematical ideas, see Katok's excellent textbook (Katok and Hasselblatt, 1997). If this mapping can be uniquely reversed, the infinite symbol sequence uniquely determines the initial value of the orbit in phase space. This can be proven by finding what is known as a "generating partition" for the iterated map. In the case of the binary shift map, for instance, the partition into two halves of the interval is such a generating partition, because the symbol sequence obtained by following an orbit beginning at x<sup>0</sup> is exactly the binary expansion of x0. For a general map it is not clear if such a partition exists, or how to find it. The practice is rather to create an equipartition of the phase space (in this case the interval), knowing that as the number of subintervals in the partition increases the accuracy of the representation increases. Here we take a coarse partition, but have limited ourselves to comparing epsilon machines created from symbol sequences from the same partition only to each other, not to any external case. This is similar to the problem of computing the entropy of a distribution with a histogram, which depends explicitly on the number of bins. Finally we note that describing orbits of iterated maps on the unit interval with a symbol sequence by partitioning the interval is common and considered to be generally applicable and advantageous if the iterates are obtained from a numerical simulation or from experimental data. This idea is taken up in Beck and Schögl (1993), and a good introductory textbook on symbolic dynamics for scientists is Lind and Marcus (1995).

# 2.4. Partition

We have considered several options for the thresholding choice. One idea would be to set the threshold at one half Pmax, differentiating between small and large responses. However, this might obscure some of the more interesting dynamics in the process, so we could make a decision based on the mean, or median of the distribution. Alternatively we can use the fixed point value for the deterministic map, which is close to the mean in low frequency cases. However, if the goal is to uncover as much of the dynamics as possible, we choose the threshold that gives a machine with the maximum statistical complexity. To do so, we computed machines for varying threshold levels in each case, computed the statistical complexity, and took the one with the largest value. We also need to make sure we were not resolving the noise in the process, which guides us to choose a threshold with care if the support of the distribution is quite small, say less than 0.2. This occurs for very low and very high frequencies typically.

See Appendix 2 for an investigation of the effect the partition has upon the resulting machine. For simpler cases finite changes in threshold do not change the topology, only the probabilities. For the facilitating case in mid-range frequencies the machine changes more dramatically as the threshold is varied. Because the statistical complexity measure quantifies the degree of structure present in the data, choosing the machine that maximizes the statistical complexity ensures that it represents the maximum structure present in the data. Then the resulting machines can be compared across the input frequency range.

#### 2.5. Machines

After partitioning, the Pr time series becomes a sequence of 0's and 1's that can be used to create HMMs. We apply the CSSR algorithm (Shalizi and Klinkner, 2004), using the Matlab package in the Causal State Modeller Toolbox (available online at http:// www.mathworks.com/matlabcentral/fileexchange/33217) (Kelly et al., 2012).

CSSR has two user-specified parameters. The significant level α, assigned by χ <sup>2</sup> or Kolmogorov-Smirnov (KS) tests, determines whether the estimated conditional distribution of histories over the next-symbol is significantly different from all of the state's other morphs. In case of a significant difference, new states are formed for these subsequences. By varying the significance level α, we control the risk of seeing structure that is not there and states merely created due to sampling error, rather than the actual differences between their morphs. Some common choices of α that work well in practice are 0.001, 0.01, 0.1, and 0.05. In our study we set α = 0.01. Also, the CSSR algorithm

this synapse model changes at this firing rate, (see Figure 1). In non-physiological range from 125 to 250 Hz, the complexity of graph structure decreases.

depends crucially on another user-set parameter, Lmax, which is the maximum subsequence length considered when inferring the model structure. It is important to find the correct value of Lmax as it defines the exponent of the algorithm complexity. Setting Lmax too large results in data shortage for long strings, the algorithm tends to produce too many states and hence the results become unreliable. On the other hand, if Lmax is too small, the algorithm won't be able to capture all statistical dependencies in the data and the state structure of the inferred machine may not be useful. Finding an optimal choice of Lmax is not straight forward. Here we determine the history length according to the relationship derived from Hanson (1993). Based on this formula, for a given number of data points N, and fixed significance level α, we choose the maximum length of subsequence L such that

$$\sqrt{\frac{|\mathcal{A}|^{L\_{\text{max}}}}{N - L\_{\text{max}}}} = \alpha.$$

where <sup>A</sup> is the alphabet size. For instance, for <sup>N</sup> <sup>=</sup> <sup>10</sup><sup>5</sup> and α = 0.01 this formula gives Lmax = 3 as a starting value. Sometimes it is still too large and another check on Lmax is whether every state in the resulting machine contains at least one sequence of that length. If not, the machine is not valid and Lmax should be decreased. For a discussion of this see (Shalizi et al., 2002). Here we have two-symbol alphabet <sup>A</sup> = {0, 1}, and we use <sup>L</sup>max <sup>=</sup> <sup>3</sup> and α = 0.01.

For cases with less complex dynamics the machine can be resolved with L = 2 (maximum of 4 causal states possible), and increasing to L = 3 gives the same result. For the more complex cases L = 3 (maximum of 8 causal states possible) was needed to capture the dynamics. In each case we checked that the machines had converged in the sense that they did not change significantly when larger data sets are considered. We also checked that the machines were well-conceived using Shalizi's rule of thumb above.

We show machines for the three different types of synapse next. What we find gives us confidence in the both the algorithm for constructing the machine, and the machine itself as a representative of the dynamics. Furthermore, we are able to use these to illustrate some of the pitfalls in relying only on histograms to elucidate the underlying dynamics of the stochastic process.

#### 3. RESULTS

Results for the depressing synapse are shown in **Figure 4** and details for all these machines can be found in Appendix 1. We indicate on the histograms where the maximum statistical complexity is with a red line. For low frequencies, the probability of getting a large Pr value (or a "1") is quite large, and its epsilon machine captures that dynamic with one state. Similarly for high frequencies the probability of getting a small Pr value (or a "0") is quite large and a one state machine results with the probabilities reversed. For intermediate frequencies, near the maximum entropy value of 2–3 Hz, the epsilon machine has 2 states, indicating a more complicated sequence of low and high Pr-values. Both 2 and 5 Hz have identical machines in structure with slight variations in the transition probabilities.

The words in each causal state indicate the kind of sequences that are typical of the synapse. For 2 and 5 Hz, state 0 contains the sequences 00, 10, 000, 010, 100, and 110. State 1 contains 01, 11, 001, 011, 101, and 111. Between the two, all possible sequences of length 2 and 3 are represented. The probability of getting a 0 or a 1 is more or less equally likely from both states. State 0 contains more zeros overall, so it is the lower Pr state. Note that the transition from state 0 to state 1 occurs with the emission of a 1, so the occurrence of a 1 in the sequence drives the dynamic to state 1, and visa versa. This is a kind of sorting of sequences into words with more zeros and those with more 1's. There is nothing particularly "hidden" in this HMM. For us it means the dynamics of the synapse are best understood in terms of the histograms. There is nothing particularly complex in the filter produced by the map.

We have already seen that the histograms for the depressing synapse are well represented by the stochastic fixed point distribution. And even though the distribution sloshes around as the frequency is varied, there is little change in complexity in the epsilon machines through this range. There are several ways to interpret this result. One is that the very short τca means there is little correlation in calcium time series, which in turn determines the correlation in P and, indirectly and directly, R. We examine this idea further in section 4. Another way is to consider the histograms themselves which are either fairly flat, or with a single peak at smaller Pr-values and an exponential type tail to the right. The structure is simple, and can be understood as a "stochastic fixed point" filter of the incoming Poisson spike train. All this is in contrast with the results for the facilitating synapse, which we show in **Figure 5**.

Histograms of the output Pr are shown in **Figures 5A–F**, for 50, 77, 100, 125, 200, 250 Hz, respectively, along with their corresponding epsilon machines of L = 3. For frequencies less than 50 Hz the machine has one state. Starting at ν = 50, all the machines can be described by referring to a persistent "inner cycle" and "outer cycle." With the exception of the 100 Hz machine, which has a third cycle, they can be related to one another by graph operations as the frequency is varied. For instance, at 50 and 77 Hz, the machines are topologically similar, with small variations in the transition probabilities. Note however that the histograms are not similar in any obvious way; the epsilon machine identifies the underlying unifying stochastic process. The outer cycle connects state 0 to 1 to 2 and back to 0. The inner cycle connects states 1 to 3 to 4 to 2 and back to 1. An additional transition exists between state 3 and 2, bipassing state 4. State 4 is notable for its self-connecting edge that emits a "1." This state also appears in all the other machines. The machine found at 125 Hz is very similar to these: the outer cycle is preserved, though now it connects states 0 to 1 to 3 and back to 0. The inner cycle can be derived from the inner cycle in the lower frequency machines by removing state 3, and sharing an edge with the outer cycle, the one connecting states 1 to 3.

The 200 Hz machine has the same inner cycle as the 125 Hz machine (connecting states 1 to 3 to 2 and back to 1, with a shared edge with the outer cycle from state 1 to state 2). The outer cycle can be made from the 125 Hz outer cycle with the addition of a state between 1 and 3 in that graph, and another edge from the new state back to 1. The machine for 250 Hz is the simplest, and can be derived from the machine at 125 Hz by merging state 1 and 2.

This leaves the most complicated structure, at 100 Hz, with 7 states. However, note that there is still an outer cycle from states 0 to 1 to 3 to 5 and back to 0. The inner cycle connects states 1 to 2 to 4 to 5 and back to 1. The third cycle runs from states 2 to 4 to 6 and back, connecting with the outer cycle at state 3. This connection gives the process another path back to state 0. The point of this rather tedious exercise is to see there is indeed an underlying structure to the overall dynamics of the synapse, with more states and transitions being revealed as the frequency is increased through 100 Hz.

The mixed synapse dynamics are surprisingly less complex, see **Figure 6**. We set the parameters of the map so that the fixed point spectrum has a local max at physiological frequencies, but apparently this can occur without creating much structure in the histograms, or complexity in the dynamics. The machines at 5, 50, and 125 Hz have 2 states and are the same as the machines at 2 and 5 Hz in the depressing case, with small variation in the transition probabilities. At 25 and 250 Hz the machines are topologically similar with different probabilities on the transitions. They have three states, and state 1 is the same as state 1 in the two state machines. States 0 and 2 together contain the sequences in state 0 of the two state machines. To move from the two state machine to the three state machine another state is added between state 1 and 0 (on that edge) and also linked back to state 1. The machine is also topologically similar to the 250 Hz machine for the facilitating synapse, though the causal states are created with L = 2 in the facilitating case.

The hierarchy of the machines for each set of parameter values is evident, and it is possible to visualize transformations of one machine into another as the firing rate is changed. To sum up these results we plot the statistical complexity of the machines as a function of frequency in each case. See **Figure 7**. We now seek to connect this back to properties of the synapse model itself.

## 4. DISCUSSION: INTERPRETATION OF RESULTS

The depressing synapse is the simplest of the three cases, and through this investigation it is clear that the formulation of the distribution of the Pr in terms of the "stochastic fixed point" gives an almost entire description of the dynamics. For very small and very large frequencies the data points are almost all 1's or 0's, respectively, so the machine has one state. In the small frequency range where the distribution slides from being concentrated at Pmax to be concentrated at zero, the epsilon machine shows that the dynamics are still simple, and can be explained by two causal states, one with mostly 0's and the other with mostly 1's. Changing frequency affects the transition probabilities on the edges only.

The other two cases are much less simple. More complicated dynamics are possible as the input firing changes. The complexity of the machines for the facilitating synapse compared to the depressing and mixed synapse can be understood by comparing the "decomposed" fixed point spectrum. See **Figure 8**. Plotting the fixed points in R and P along with Pr shows a striking difference between the three cases. The depressing synapse Pr fixed point is entirely controlled by the variation in R, as P

remains constant over the range of frequencies, and the P variation happens in a very small range of frequencies near zero. The facilitating synapse has a range of R-values across the spectrum, as well as a range of P-values. The mixed synapse has a very little variation in P. The more complicated machines in both cases occur at frequencies where there is the largest variation in both. Obviously, having a range of response in both P and R creates the complexity of the machines, however indirectly.

Another way to view this difference is through the calcium decay time. For the depressing synapse τca is very short, and there is very little correlation in the calcium time series in all but very high frequencies (which are not physiological). The synapse simply filters the Poisson spike train process. In the mixed case, while the calcium time series is more correlated, the lack of variation of the P response flattens out any downstream effect on Pr. The facilitating case is really in the "goldilocks zone" where the correlation in the calcium time seriescan effect Pr through the variation in P. A synapse might be expected to be a more complex filter if there is a hidden time dependent variable, such as calcium, that links the two processes of facilitation and depression, as it does in this model for larger τca or higher frequencies. The exact details of the relationship between the probability of release, and the rate of recovery of R as they depend upon C must line up to produce sensitivity in the fixed point values for each in the same frequency range.

Finally, we note that the histograms themselves, from which many information measures are constructed, do not tell the whole story. There is a much more complicated dynamic occurring in the facilitating synapse than the depressing synapse, though comparing the histograms themselves in the two cases does not suggest this. We have also seen the converse, where the machines are the same, even though the distributions are quite different. This implies that both are needed to have a full understanding of such a stochastic process.

# 5. CONCLUSIONS

In this paper we demonstrate the validity of using causal state models to more completely describe stochastic short-term synaptic plasticity. These models rely only upon output data from a synaptic connection, knowledge of the input stimulus stream is not required. This will expand the arena of experiments where data can directly inform models, and more importantly uses the data itself to create models. While these models are not physiologically motivated per se, we have shown how we can connect the structure of the model to complexity of the mechanisms involved, a useful first step in a more complete categorization of short term plasticity. Interpreting synaptic plasticity in the language of computation could also be exploited in the construction of large scale models of neural processes involving many thousands of neural connections, and potentially lead to a more complete theoretical description of the computations possible.

Our results also draw direct connections between the causal state models and the deterministic dynamics of the underlying model used to create the data. Specifically, they point to the importance of having variability in both probability of release and the recovery rate of resources with frequency in creating a more complex synaptic filter. This finding can be reversed (at some peril, we realize) to imply that a more complicated machine results from a synapse with such variability. This in turn could be used to inform the development of physiologically accurate models, or direct future experimental design. Interested reader may receive any/all of the code use to create these results by contacting Elham Bayat-Mohktari.

# 6. FUTURE WORK

The model of the synapse we used to create the data was parameterized from experimental data from an actual depressing synapse in the hippocampus. The experiments gave the synapse uniformly spaced stimuli at fixed frequencies. Our work suggests that a more comprehensive understanding of the dynamics of the synapse could be found by using a predetermined stochastic input, such as a Poisson spike train. The distributions of the responses could then be fit if the desire was to estimate parameters of an a priori model. This fitting could be done using Bayesian techniques as well as standard statistical methods.

The other approach would be to let the data from such an experiment create the model itself, in the form of epsilon machines or perhaps some other form of HMM. We have seen here that the machine reconstruction process can be used for classification purposes, and can uncover features not obvious from the distributions of the response. It is also possible to describe such short term synaptic plasticity as a simple computing operation, or Turing Machine (Copeland, 2004) but the graph model of this is not unifilar, so making a simple connection between it and epsilon machines, or creating a nonunifilar HMMs from data, are topics for further investigation.

Finally, describing the evolution of one epsilon machine to another as a parameter is varied in terms of graph operations could give one more description of an entire range of behavior of a short term plasticity filter, as a parameter is varied. We are currently investigating this approach.

# AUTHOR CONTRIBUTIONS

EB did all the statistical and computational mechanics analyses in the study; JL carried out all experiments and preliminary data analysis; ES conceived of the study, developed the design, and analyzed the results. All authors read and approved the final manuscript.

# FUNDING

Electrophysiology experiments were performed in the laboratory of Chris McBain with intramural support from National Institute of Child Health and Human Development. Later work was supported by National Center for Research Resources Grant P20-RR-015583, National Institutes of Health Grant R01069689- 01A1, and start-up support from the University of Montana Office of the Vice President for Research (to JL).

# ACKNOWLEDGMENTS

We acknowledge David Patterson for his helpful comments on some of the statistical techniques.

#### REFERENCES


# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncom. 2018.00032/full#supplementary-material

at parvalbumin-positive basket cell synapses from juvenile ca1 mouse hippocampus. J. Neurophysiol. 113, 2408–2419. doi: 10.1152/jn.00167.2014


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Bayat Mokhtari, Lawrence and Stone. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Image Processing for Bioluminescence Resonance Energy Transfer Measurement—BRET-Analyzer

Yan Chastagnier 1, 2, 3, Enora Moutin1, 2, 3, Anne-Laure Hemonnot 1, 2, 3 and Julie Perroy 1, 2, 3 \*

<sup>1</sup> Centre National de la Recherche Scientifique, UMR-5203, Institut de Génomique Fonctionnelle, Montpellier, France, 2 Institut National de la Santé Et de la Recherche Médicale, U1191, Montpellier, France, <sup>3</sup> Universités de Montpellier, UMR-5203, Montpellier, France

A growing number of tools now allow live recordings of various signaling pathways and protein-protein interaction dynamics in time and space by ratiometric measurements, such as Bioluminescence Resonance Energy Transfer (BRET) Imaging. Accurate and reproducible analysis of ratiometric measurements has thus become mandatory to interpret quantitative imaging. In order to fulfill this necessity, we have developed an open source toolset for Fiji—BRET-Analyzer—allowing a systematic analysis, from image processing to ratio quantification. We share this open source solution and a step-by-step tutorial at https://github.com/ychastagnier/BRET-Analyzer. This toolset proposes (1) image background subtraction, (2) image alignment over time, (3) a composite thresholding method of the image used as the denominator of the ratio to refine the precise limits of the sample, (4) pixel by pixel division of the images and efficient distribution of the ratio intensity on a pseudocolor scale, and (5) quantification of the ratio mean intensity and standard variation among pixels in chosen areas. In addition to systematize the analysis process, we show that the BRET-Analyzer allows proper reconstitution and quantification of the ratiometric image in time and space, even from heterogeneous subcellular volumes. Indeed, analyzing twice the same images, we demonstrate that compared to standard analysis BRET-Analyzer precisely define the luminescent specimen limits, enlightening proficient strengths from small and big ensembles over time. For example, we followed and quantified, in live, scaffold proteins interaction dynamics in neuronal sub-cellular compartments including dendritic spines, for half an hour. In conclusion, BRET-Analyzer provides a complete, versatile and efficient toolset for automated reproducible and meaningful image ratio analysis.

Keywords: open source software, automatic image analysis, ratiometric measurements, bioluminescence resonance energy transfer

# INTRODUCTION

A perpetual association and dissociation between proteins drives specific cellular signaling in time and space. Bioluminescence Resonance Energy Transfer (BRET) imaging is a sensitive technology to highlight the spatio-temporal dynamics of protein-protein interaction and understand their functions in intact living cells (Coulon et al., 2008; Perroy, 2010; Goyet et al., 2016; Faklaris et al., 2017). Briefly, the principle of the method stands on an energy transfer between a bioluminescent

#### Edited by:

Jean-Marie Charles Bouteiller, University of Southern California, United States

#### Reviewed by:

Karim Nagi, Duke University Medical Center, United States Terence Hébert, McGill University, Canada

> \*Correspondence: Julie Perroy Julie.perroy@igf.cnrs.fr

Received: 24 November 2017 Accepted: 20 December 2017 Published: 09 January 2018

#### Citation:

Chastagnier Y, Moutin E, Hemonnot A-L and Perroy J (2018) Image Processing for Bioluminescence Resonance Energy Transfer Measurement—BRET-Analyzer. Front. Comput. Neurosci. 11:118. doi: 10.3389/fncom.2017.00118 donor and a compatible fluorescent acceptor (Xu et al., 1999; Angers et al., 2000). By catalytic oxidation of its substrate, the bioluminescent donor emits light (donor image, D, **Figure 1**). Upon molecular proximity (<75Å), the BRET compatible acceptor is excited by a non-radiative transfer of energy and in turn emits light at its characteristic wavelength (acceptor image, A, **Figure 1**). Pixel by pixel division of the light emitted by the acceptor over the light emitted by the donor (A/D) gives rise to the BRET image, a ratiometric measurement expressed as pseudo-colors, allowing live quantification of the interaction between proteins tagged with BRET compatible entities, in subcellular domains (Goyet et al., 2016). The advantage of BRET over other RET methodologies precisely comes from the absence of light to initiate the energy transfer in BRET. Thereby, BRET circumvents many drawbacks linked to light excitation (such as auto-fluorescence of cells, direct excitation of the acceptor fluorophore by the donor exciting light, or photobleaching of fluorophores) giving rise to an excellent signal over noise ratio.

A ratiometric measurement provides the huge benefit to normalize the output signal to the intensity of its stimulus. Hence, any efficient activation of a system can virtually be reported, regardless of the stimulus intensity. This is particularly relevant to put emphasis on effective forces of small ensembles. However, this benefit is counterbalanced by the fact that a nearbackground stimulus will give rise to an aberrant ratiometric measure resulting from the fraction's near-zero denominator. Consequently, the specific signal is lost in a high-intensity background. Therefore, delimitation of the linear detection range of the recording system and proper determination of the stimulus intensity threshold is mandatory to perform ratiometric measurements. In the present work, we have developed a systematic image processing to obtain relevant ratiometric measurements. We applied this automatic processing on BRET images. This homemade toolset is a complete open source solution for Fiji [free software for scientific image analysis, (Schindelin et al., 2012)], available for the scientific community together with a step-by-step tutorial (https://github. com/ychastagnier/BRET-Analyzer).

The BRET-Analyzer toolset includes classical processing of the D and A images by cleaning the background, and pixel by pixel division of the A/D images. We worked mainly on two parameters. First we defined a donor threshold computed as a function of the donor signal intensity in a given area. This step allows defining the frontiers between the sample luminescence per se and the light spread on neighboring pixels. Second we selected and combined different threshold processes independently in each sub-area, depending on the local donor intensity. This is particularly relevant when the recorded cell displays heterogeneous subcellular volumes. For example in neurons, a cytosolic signal from the donor accumulated in the soma will be much brighter than the donor light recorded in thin neuritic processes. A combination of donor threshold processing allows proper reconstitution of the ratiometric image. It is fundamental to note that getting rid of low-level donor pixels permits the selection of pixels from the luminescent specimen only, but does not influence the ratiometric measurement per se. We thus provide here a toolset—BRET-Analyzer—that allows

FIGURE 1 | Principle of the BRET-Analyzer toolset. The BRET-Analyzer toolset provides automatic analyses of donor (D) and acceptor (A) images (A) to generate a quantitative analysis of the specific A/D ratio image (B). (A) Scheme of two BRET donor compatible entities emission spectra: Nanoluciferase (Nluc) is the BRET donor and Venus the BRET acceptor. The light emitted by the donor alone displays an emission pic at 450 nm (blue curve). When Venus is co-expressed and close enough (<75Å) from Nluc, a non-radiative transfer of energy from Nluc excites Venus, which emits light at its characteristic wavelength (pic at 535 nm, green curve). Light collected with specific filters gives rise to donor and acceptor images. (B) Distribution of the A/D ratio pixel values and possible representations of the BRET images: A stack histogram (left panel) displays the distribution of the A/D ratio values obtained from pixel by pixel division of the A image over the D image, suggesting the minimal and maximal values to be set on the pseudo-color scale expressing the A/D ratio image. "Raw," "weighted" and "specific" panels are three possible representations of the A/D ratio image. The "raw" image expresses the ratio as pseudo-colors, without any selection of pixels. Consequently, pixels with a near-zero intensity on the D image display a maximal A/D ratio, compromising the detection of pixels from the specimen. Alternatively, the "weighted" ratio also displays the A/D ratio intensity as pseudocolors, but the brightness of each pixel is proportional to the donor intensity. This representation shades the ratio background around the cell but also the specific signal coming from low protein expression level in sub-cellular area. The BRET-Analyzer toolset allows obtaining the "specific" ratio by an automatic selection of pixels from the specimen only (obtained by composite thresholding of the D image). The "specific" image displays the intensity of the A/D ratio expressed as pseudo-colors, regardless of protein expression levels.

the elimination of non-specific signal, and performs quantitative analysis of long period recordings of protein-protein interactions, regardless of the protein expression level (**Figure 1**).

# STEPWISE PROCEDURES AND ANTICIPATED RESULTS

Raw images of the donor (D) and acceptor (A) BRET entities were obtained as previously described (Goyet et al., 2016), from hippocampal neurons (**Figure 1**). To summarize the process, the images will first have their background subtracted, then be aligned over time. A threshold will be computed to separate signal from background. A image is divided by D image pixel by pixel. Finally, measures are made on the ratiometric images obtained.

The analysis starts with the removal of non-specific signals ("clean" button, Supplementary Material 4.1), in D and A images, in 3 simple steps. First, in each image, applying a median filter of radius 1 (3∗ 3 pixels) allows to remove outliers. Second, subtracting the median value of a region corresponding to the background removes the camera offset and global light background, assuming they are homogeneous. The homogeneous assumption is based on the fact that BRET doesn't use illumination, unlike fluorescence based imaging. Subtracting a background image is available as an option. This value is measured on each image of the donor and the acceptor, to account for potential fluctuations of the background's mean intensity from an image to another. Third, in case recording is a timelapse, aligning images might be needed. To do so, we make use of the plugin TurboReg (http://bigwww.epfl.ch/thevenaz/ turboreg/ Thevenaz et al., 1998), a pyramid approach to subpixel registration based on intensity. This alignment can rescue xy drift that may happen during multi-positioning of the microscopy setup to record distant cells on the same sample. It is important that images are aligned through time to make sure the area in which the measures are made always corresponds to the same part of the sample.

Second step is to divide, pixel by pixel, the A image by the D image ("divide" button, Supplementary Material 4.3). Prior to this division, it is necessary to get rid of all pixels that have an intensity level too close to the noise level on the donor image, to keep only specific signal from the sample (**Figure 1B**, "specific"). Indeed, the computed A/D ratio tends to infinity for each pixel with a near-zero intensity on the D image. If these pixels are not removed, the ratiometric values can range from zero to hundreds and the boundaries of the cells cannot be distinguished (**Figure 1B**, "raw"). Alternatively, the brightness of the A/D ratio image can be weighted by the expression level of proteins (**Figure 1B**, "weighted"), which efficiently hides non-specific pixels, but also fades ratiometric values from small protein numbers. This weighting can be attractive for homogenous samples but should be avoided to perform ratio-metric measurements from heterologous volume specimens since it overshadows proficient strengths from small ensembles.

Hence, in order to remove pixels that are outside of luminescent specimen from the resulting A/D ratio image, we applied available threshold methods to the D image (**Figure 2**). The simplest approach, applying a static threshold level of a given value across all images (**Figure 2B**) presented two main problems. First, since the D signal decreases with time, part of the specific D signal falls below the threshold during image time series when the threshold was determined on the first image. Conversely, lowering the threshold resulted in keeping the noise on the first images of the image time series. We circumvented this problem by computing a specific threshold for each individual image of the image time series. We selected a region of interest (which should contain about half pixels to keep and half to remove). For each time point, we computed the median of that area as the threshold to be applied on the D image (**Figure 2C**). Alternatively, we used an automatic threshold built in Fiji. Across all the available automatic thresholding methods, Otsu [which searches for the threshold value that minimizes the intra-class variance (Otsu, 1979)] gave the best results at separating the high signal from the rest (**Figure 2D**).

However, a second problem was related to the fact that, for a given time point, the D image displayed strong intensity differences among neuronal subareas (**Figure 2A**). Therefore, applying a homogenous constant threshold on the D image did not allow suppressing pixels adjacent to the neuronal cell body (which, even so located out of the neuron, still displayed a relatively high light intensity due to the D light spread), while keeping the low but specific D light from neuritic processes (**Figures 2B–D**). To solve this spatial problem, one possibility was to crop different regions of interest ("crop" button, Supplementary Material 4.2) and compute a specific threshold for each crop. But the resulting BRET image of the neuron was segmented in several areas. Furthermore this process increased analysis time to quantify the BRET images (step below), as it required a repeated exploration on each crop. We thus favored a second analysis process to determine the threshold of the D image in space. Automatic local thresholds built in Fiji have the advantage to compute the threshold for each pixel based on the surrounding pixels. Hence values that are locally high are selected, allowing inhomogeneous signal across the image. For example Phansalkar threshold (Phansalskar et al., 2011), provided good results at removing noise at the edges of cells making clear delimitation, but kept a good part of the noise far from the cells (**Figure 2E**).

We thus designed our own composite thresholding method (Chastagnier threshold, **Figure 2F**). The whole process consisted in drawing an accurate mask around the neuron to keep only pixels of the D image arising from the luminescent specimen. To create this mask (**Figure 3**, gray rectangle), we first subtracted a blurred version of the D image to itself in order to increase the contrast, and convert it to binary values using Li's automatic threshold method (Li and Tam, 1998). When the blur effect is decreased, noise areas start to show up, while when blur effect is increased, areas of interest show up thicker than they really are. To get rid of the defects of both low and high blur effect, we used both and combined them with a logical AND in order to keep only pixels that are in both images. As "holes" appeared in the lowest intensity area of two juxtaposed regions of high intensity areas, we combined it using a logical OR with the original image on which Otsu Threshold was applied (to keep only high signal). Logical OR is inclusive, so it keeps pixels that are in an image, in the other, or in both. We thus obtained a binary image with values 0 and 255. Dividing it by 255 gave 0 for pixels we want to remove and 1 for pixels to keep. This is the mask (**Figure 3**, Chastagnier Threshold). Multiplying it with the original D image removed irrelevant pixels (set them to 0) and did not affect the value for the rest of the D image (**Figure 3**, Thresholded donor). The Chastagnier's threshold thus unequivocally refined the limits of the luminescent sample in space and time. For user convenience, we nevertheless included the conventional

(background around the luminescent specimen) in the A/D ratio image. Various threshold methods (B–F) can be applied to the donor image (A). Each column represents a different time point. (A) Intensity of the donor for each time point. (B) Mask using a constant threshold all over the image time series. (C,D) Mask using a global threshold computed for each time point (median threshold of the area outlined in red, (C) or Otsu thresholding method, (D). (E,F) Mask using a local threshold computed for each time point, Phansalkar (E) or Chastagnier threshold (F), described in Figure 3. Note that computing a local threshold for each image over the time series allows to precisely define the luminescent specimen limits, all along the experiment.

alternative thresholding methods herein tested, as an option in the BRET-Analyzer toolset.

The A image was then divided by the thresholded D image and displayed with a 16 pseudo-colors look up table, ranging from cold to hot colors (Supplementary Material 4.3.5). The minimal and maximal A/D ratio values between which the colors will be distributed have to be selected. The stack histogram displayed between the 5th and 995th permilles (without taking into account pixels with value 0) can be used to help the user choose appropriate values (**Figure 1B**). Once the range is set, it should be kept the same for every image, so they can be visually compared, before doing the in depth analysis. The resulting BRET image is a quantitative measurement of the energy transfer intensity reporting either the efficiency of interaction between two proteins tagged with BRET compatible entities (intermolecular BRET) or conformational changes of a molecule tagged with D and A entities (intramolecular BRET). This analysis displays BRET intensity regardless of the level of protein expression. Hence, even small numbers of complexes are measured and visualized on the BRET image. This provides an important benefit compared to ratiometric analysis in which the signal is expressed as discontinuous pseudo-color scale representing two different parameters in one, namely A/D ratio and expression level (the brightness of the image being weighted by the protein expression level, e.g., by the D and/or A image intensity). This is potentially confusing and may lead to arbitrary judgments about the extent of BRET. Indeed, this kind of analysis, presented on **Figure 1** as "weighted" A/D ratio allows to visually exclude the noise but the evidenced BRET signals come from proteins with high expression level only, neglecting the functional importance of high protein interactions of small ensembles. The weighted representation is nevertheless included as an option in the BRET-Analyzer (Supplementary Material 4.5 Param Tool, Display weighted images?).

Final step is to extract the BRET values out of the ratiometric image, in regions of interest, which can be chosen

on a single image (**Figure 4**), or pooled from multiple images (Supplementary Material 4.4). All tools can be used to define the area of interest (rectangle, polygon, freehand. . . ), at user convenience. It is not necessary to precisely draw the limits of the samples with the tool, because all pixels with a null value are being automatically ignored by the measures and excluded

FIGURE 5 | hour. By opposition, the conventional analysis failed to quantify the BRET in spines over time. (B) BRET-Analyzer enables live, BRET image time series of mGlu5-Homer interaction in neurons. We quantified the mean BRET intensity in single areas drawn on soma and dendritic spines over time, before and during neuronal depolarization (KCl 55 mM). KCl 55 mM decreases BRET signals in soma and spines. BRET-Analyzer enables precise measurements of ratiometric signals over time. Each point of the curves is the mean ± SEM of BRET intensity recorded in 3 areas.

from the analysis of the BRET quantification in the area. The mean BRET intensity provides a quantification of the number of protein complexes in a given area and the standard deviation of the BRET intensity between pixels reports homogeneity or clusterization of the BRET signal.

In addition to the automation of analysis process, the main strength of BRET-Analyzer resides in the potency to quantify luminescent ratio regardless of protein expression levels. This is particularly relevant when working on heterogeneous subcellular volumes such as neurons. Indeed, a neuron contains a large range of volumes from the neuronal cell body (around 500 µm<sup>3</sup> ) to small dendritic spines (spine head volumes ranging from 0.01 µm<sup>3</sup> to 0.8 µm<sup>3</sup> ). The first challenge, as discussed before is thus to find an adequate composite threshold allowing to perform ratiometric measurements in all volumes at the same time. Moreover for the smallest compartments, measuring protein-protein interactions from few proteins is difficult and live dynamics over time are even more challenging. To illustrate the benefits from the herein developed analysis toolset, we analyzed twice the same images either with the conventional analysis (including a static threshold level of D images, described in **Figure 2B**) or with BRET-Analyzer (using Chastagnier's threshold, **Figure 2F**). We thus measured in neurons the interaction between a glutamate receptor, mGlu5, and its cognate scaffold, Homer, together involved in neuronal synaptic transmission (Moutin et al., 2012; Guo et al., 2015). As shown in **Figure 5A**, the interaction between mGlu5 and Homer was accurately recorded over time in the soma and the quantification indiscernible whatever was the analysis protocol. However, the advantage of using BRET-Analyzer became obvious for small processes, which from the beginning of the experiment were barely identifiable using the classical analysis protocol and totally lost few minutes after. BRET-Analyzer enables long time recording of BRET signals in small subcellular volumes, allowing for the first time live and stable measurements of mGlu5-Homer interaction in individual spines for half an hour. By opposition to previous studies (Moutin et al., 2012; Guo et al., 2015), instead of assessing synaptic activity-induced changes in mGlu5-Homer interactions in spines from two distinct cell populations (stimulated or not), we could here follow BRET changes using BRET-Analyzer in the same spines before and during KCl-induced neuronal depolarization. Thus, neuronal depolarization disrupted mGlu5-Homer interaction (**Figure 5B**). We recorded a 15 and 16% drop of BRET signal in soma and spines, respectively. This BRET decrease was measured as soon as 2 min following KCl 55 mM perfusion and stable for 20 min. Hence, BRET-Analyzer enables image time series of ratiometric measurements even in small cellular compartments.

# MATERIALS

#### Source Code

The data were analyzed using a toolset made for Fiji (https:// fiji.sc) (Schindelin et al., 2012). We here provide a stepby-step tutorial as Supplementary Material. The toolset and documentation details of how to use it are also publicly available at (https://github.com/ychastagnier/BRET-Analyzer), and will be regularly updated. Moreover, the tools can be freely downloaded, modified and improved to fit future research needs.

TurboReg, the plugin used to align stacks of images can be downloaded at (http://bigwww.epfl.ch/thevenaz/turboreg/) (Thevenaz et al., 1998).

### Example Luminescent Images

The donor and acceptor fluorescent images used here to exemplify the use of BRET-Analyzer corresponds to an hippocampal neuron from primary cell culture transfected with an intramolecular BRET-based sensor for ERK activity [YEN (Goyet et al., 2016)], except in **Figure 5**, where mGlu5-NanoLuc and Venus-Homer intermolecular BRET was recorded. Raw images are available at https://github.com/ychastagnier/BRET-Analyzer.

# AUTHOR CONTRIBUTIONS

YC designed the Fiji BRET-Analyzer plugin, computed all steps of the automated image analysis protocol and wrote the manuscript and step by step tutorial. EM and A-LH performed BRET imaging experiments. JP conceived, designed, led the project, supervised the analysis strategy and wrote the manuscript.

# ACKNOWLEDGMENTS

We are grateful to Vincent Compan, Laurent Fagni, Elise Goyet, and Vincent Ollendorff for helpful reading of the manuscript. This work was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (JP, grant agreement No. 646788), the Agence Nationale de la Recherche (JP, ANR-13-JSV4-0005-01) and the Région Languedoc-Roussillon (JP, Chercheur d'Avenir 146090).

### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncom. 2017.00118/full#supplementary-material

# REFERENCES


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Chastagnier, Moutin, Hemonnot and Perroy. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Modulation of Spike-Timing Dependent Plasticity: Towards the Inclusion of a Third Factor in Computational Models

#### Alexandre Foncelle1,2 \* † , Alexandre Mendes 3,4 \* † , Joanna Je˛ drzejewska-Szmek 5‡ , Silvana Valtcheva3,4 ‡ , Hugues Berry 1,2\*, Kim T. Blackwell <sup>5</sup> \* and Laurent Venance3,4 \*

1 INRIA, Villeurbanne, France, <sup>2</sup>LIRIS UMR 5205 CNRS-INSA, University of Lyon, Villeurbanne, France, <sup>3</sup>Dynamic and Pathophysiology of Neuronal Networks, Center for Interdisciplinary Research in Biology (CIRB), College de France, INSERM U1050, CNRS UMR7241, Labex Memolife, Paris, France, <sup>4</sup>University Pierre et Marie Curie, ED 158, Paris, France, <sup>5</sup>The Krasnow Institute for Advanced Studies, George Mason University, Fairfax, VA, United States

In spike-timing dependent plasticity (STDP) change in synaptic strength depends on the timing of pre- vs. postsynaptic spiking activity. Since STDP is in compliance with Hebb's postulate, it is considered one of the major mechanisms of memory storage and recall. STDP comprises a system of two coincidence detectors with N-methyl-D-aspartate receptor (NMDAR) activation often posited as one of the main components. Numerous studies have unveiled a third component of this coincidence detection system, namely neuromodulation and glia activity shaping STDP. Even though dopaminergic control of STDP has most often been reported, acetylcholine, noradrenaline, nitric oxide (NO), brain-derived neurotrophic factor (BDNF) or gamma-aminobutyric acid (GABA) also has been shown to effectively modulate STDP. Furthermore, it has been demonstrated that astrocytes, via the release or uptake of glutamate, gate STDP expression. At the most fundamental level, the timing properties of STDP are expected to depend on the spatiotemporal dynamics of the underlying signaling pathways. However in most cases, due to technical limitations experiments grant only indirect access to these pathways. Computational models carefully constrained by experiments, allow for a better qualitative understanding of the molecular basis of STDP and its regulation by neuromodulators. Recently, computational models of calcium dynamics and signaling pathway molecules have started to explore STDP emergence in ex and in vivo-like conditions. These models are expected to reproduce better at least part of the complex modulation of STDP as an emergent property of the underlying molecular pathways. Elucidation of the mechanisms underlying STDP modulation and

#### Edited by:

Jean-Marie Charles Bouteiller, University of Southern California, United States

#### Reviewed by:

Tomoki Fukai, RIKEN Brain Science Institute (BSI), Japan Paul Miller, Brandeis University, United States

#### \*Correspondence:

Hugues Berry hugues.berry@inria.fr Kim T. Blackwell kblackw1@gmu.edu Laurent Venance laurent.venance@college-de-france.fr

†Co-first authors.

#### ‡Present address:

Joanna Je˛ drzejewska-Szmek, Department of Neurophysiology, Nencki Institute of Experimental Biology, Warsaw, Poland Silvana Valtcheva, Departments of Otolaryngology, Neuroscience and Physiology, Skirball Institute, Neuroscience Institute, New York University School of Medicine, New York, NY, United States

> Received: 15 March 2018 Accepted: 06 June 2018 Published: 03 July 2018

#### Citation:

Foncelle A, Mendes A, Je˛ drzejewska-Szmek J, Valtcheva S, Berry H, Blackwell KT and Venance L (2018) Modulation of Spike-Timing Dependent Plasticity: Towards the Inclusion of a Third Factor in Computational Models. Front. Comput. Neurosci. 12:49. doi: 10.3389/fncom.2018.00049

**Abbreviations:** AMPAR, α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptor; ATP, adenosine triphosphate; BDNF, brain-derived neurotrophic factor; cAMP, cyclic adenosine monophosphate; CaMKII, Ca2+/calmodulin-dependent protein kinase-II; CB1R, cannabinoid type-1 receptor; cGMP, cyclic guanosine monophosphate; DARP-32, dopamineand cAMP-regulated phosphoprotein, Mr 32 kDa; DXR, dopaminergic type-X receptor; EAAT2, excitatory amino acid transporter-2; ERK, extracellular signal–regulated kinase; GABA, gamma-aminobutyric acid; ITDP, input-timing dependent plasticity; mAChRs, muscarinic acetylcholine receptors; mGluR, metabotropic glutamatergic receptor; MX, muscarinic type-X receptor; nAChRs, nicotinic acetylcholine receptors; NMDAR, N-methyl-D-aspartate receptor; NO, nitric oxide; STDP, spike-timing dependent plasticity; tLTD, timing-dependent long term depression; tLTP, timing-dependent long term potentiation.

its consequences on network dynamics is of critical importance and will allow better understanding of the major mechanisms of memory storage and recall both in health and disease.

Keywords: STDP, third factor, dopamine, acetylcholine, noradrenaline, astrocytes, eligibility traces, Hebbian plasticity

#### INTRODUCTION

Most computational and experimental studies of synaptic plasticity focus on variations of Hebb's rule in which the change in synaptic strength is caused by direct association of two factors, i.e., two inputs (or activity patterns), one on the presynaptic and one on the postsynaptic side. Thus, when neural circuits adjust their synaptic weights depending on the frequency or timing of the pre-synaptic and post-synaptic firing patterns, Hebb's postulate is fulfilled. In addition, a third factor (for example neuromodulators or astrocytes) stabilizes or modulates the expression of synaptic plasticity and, thus, ultimately learning (Kempter et al., 1998; Pawlak et al., 2010; Lisman et al., 2011; Frémaux and Gerstner, 2016; Edelmann et al., 2017; Ku´smierz et al., 2017; Gerstner et al., 2018). The inclusion of this third factor with two-factor Hebbian plasticity rule is called neoHebbian plasticity (Lisman et al., 2011), and is infrequent in computational models of spike-timing dependent plasticity (STDP). In this review article, we focus on STDP (Sjöström et al., 2008; Feldman, 2012), a synaptic Hebbian learning rule, and its control by the third factor: neuromodulation (via the action of dopamine, acetylcholine, noradrenaline and others) or astrocyte activity. Our goal is to highlight aspects of STDP that should be taken into account in future computational models of STDP.

Since its discovery, STDP has attracted considerable interest in experimental and computational neuroscience because it avoids implausibly high firing frequencies and instead relies on spike correlation. STDP has emerged as a candidate mechanism for experience- and activity-dependent changes in neural circuits, including map plasticity (Abbott and Nelson, 2000; Dan and Poo, 2006; Morrison et al., 2008; Sjöström et al., 2008; Feldman, 2012; Froemke, 2015). Experiments in different brain regions and in diverse neuronal types have revealed a plethora of STDP forms that vary in plasticity direction, temporal dependence and the involvement of signaling pathways (Sjöström et al., 2008; Feldman, 2012; Korte and Schmitz, 2016). Experimental protocols that investigate STDP use pairing of a presynaptic stimulation with a postsynaptic spike, with the pre- and postsynaptic stimulations separated by a fixed interval ∆tSTDP(spike timing). In most of the studies, the spike timing is computed as ∆tSTDP = tpost−tpre, where tpost and tpre are the times of emission of the postsynaptic spike and that of the presynaptic stimulation, respectively. If the postsynaptic stimulation occurs before the presynaptic, ∆tSTDP < 0 (post-pre pairings), whereas ∆tSTDP > 0 when the presynaptic stimulation occurs before the postsynaptic one (pre-post pairings). The same pairing pattern is then repeated between 50 and 200 times at a constant frequency (typically between 0.1 Hz and 5 Hz). The canonical STDP is bidirectional (able to generate potentiation and depression depending on the value of ∆tSTDP) and Hebbian, i.e., postpre pairings (∆tSTDP < 0) yield timing-dependent long-term depression (tLTD) and pre-post pairings (∆tSTDP > 0) give rise to timing-dependent long-term potentiation (tLTP). For most STDP forms, the expression of plasticity is restricted to a narrow temporal window (|∆tSTDP| < 80 ms); thus, when pre- and postsynaptic activities are separated by a large ∆tSTDP, long-term synaptic changes are not observed (Markram et al., 1997; Bi and Poo, 1998).

The predominant form of STDP is Hebbian, and has been observed in the neocortex (Markram et al., 1997; Feldman, 2000; Sjöström et al., 2001; Froemke et al., 2005; Nevian and Sakmann, 2006), the hippocampus (Debanne et al., 1997, 1998; Bi and Poo, 1998; Nishiyama et al., 2000; Wittenberg and Wang, 2006), and the striatum (Fino et al., 2008, 2009; Pawlak and Kerr, 2008; Shen et al., 2008). In contrast to Hebbian STDP, bidirectional anti-Hebbian STDP expresses tLTP for ∆tSTDP < 0 and tLTD for ∆tSTDP > 0. Anti-Hebbian STDP was first reported in the cerebellum-like structure of electrical fish (Bell et al., 1997). More recently, bidirectional anti-Hebbian STDP has been observed in mammals and in various structures including the striatum (Fino et al., 2005, 2010; Schulz et al., 2010; Paille et al., 2013; Valtcheva et al., 2017) and the somatosensory cortex (Letzkus et al., 2006). Unidirectional anti-Hebbian forms of STDP inducing tLTD for both ∆tSTDP < 0 and ∆tSTDP > 0, have been observed in the cerebellum (Han et al., 2000; Safo and Regehr, 2008), the neocortex (Egger et al., 1999; Lu et al., 2007), the dorsal cochlear nucleus (Tzounopoulos et al., 2004) and the hippocampus (Wittenberg and Wang, 2006). Recently, a unidirectional Hebbian STDP where tLTP was observed for both post-pre and pre-post pairings, has been reported in hippocampus (Mishra et al., 2016). The mechanisms that produce these diverse forms of STDP are not completely understood, though could involve a third factor, such as neuromodulators (such as dopamine or acetylcholine; for reviews see Pawlak et al., 2010; Edelmann et al., 2017) or astrocytes.

All the forms of STDP described so far depend on one of three main systems of coincidence detectors (Feldman, 2012; Korte and Schmitz, 2016). The first system comprises the N-methyl-D-aspartate receptor (NMDAR) receptor (NMDAR) as the unique coincidence detector for both tLTP and tLTD, though voltage-sensitive calcium channels may play a role in coincidence detection. This form of plasticity has been reported in hippocampal CA1 neurons (Nishiyama et al., 2000), neocortical layer 2/3 pyramidal cells (Froemke et al., 2005), striatal output neurons (Pawlak and Kerr, 2008) and striatal gamma-aminobutyric acid (GABA)ergic interneurons (Fino et al., 2008). The second system combines NMDAR-dependent tLTP with tLTD which depends on metabotropic glutamate receptor (mGluR)- and/or cannabinoid type-1 receptor (CB1R) activation. Though the tLTD is independent of postsynaptic NMDARs, the activation of presynaptic NMDARs can be implicated (Sjöström et al., 2003; Bender et al., 2006; Corlew et al., 2007; Rodríguez-Moreno and Paulsen, 2008). This form of plasticity has been observed in the visual (layer 2/3) and somatosensory (layer 5) cortex (Sjöström et al., 2003; Bender et al., 2006; Nevian and Sakmann, 2006; Corlew et al., 2007; Rodríguez-Moreno and Paulsen, 2008), cholinergic striatal interneurons (Fino et al., 2008) or striatal output neurons (Fino et al., 2010). Recently in striatal output neurons, a third system has been reported, in which the tLTD is CB1R-dependent, whereas the molecular dependence of tLTP is governed by the number of pairings: a small number of pairings (∼10) produces a CB1R-mediated tLTP, whereas greater number of pairings yields an NMDAR-mediated tLTP (Cui et al., 2015, 2016).

The molecular mechanisms accounting for these various forms of STDP are not yet fully understood, despite a substantial number of studies focusing on STDP. For the NMDARdependent tLTP and tLTD, calcium amplitude seems to partly determine plasticity direction (Nevian and Sakmann, 2006). For ∆tSTDP > 0, when the presynaptic activity precedes the back-propagating action potential, the excitatory post-synaptic potential coincides with the back-propagating action potential resulting in high and more prolonged calcium influx through the NMDAR and voltage-sensitive calcium channels, which leads to tLTP. For ∆tSTDP < 0, calcium influx through the NMDARs and voltage-sensitive calcium channels is lower and as a result induces tLTD (Magee and Johnston, 1997; Koester and Sakmann, 1998; Nevian and Sakmann, 2006; Pawlak and Kerr, 2008). These different calcium dynamics produce different directions of plasticity by recruiting different downstream signaling molecules. Several computational models have used a description of neuronal calcium dynamics and/or the kinetics of downstream signaling pathways as a proxy to predict the direction of plasticity (tLTP or tLTD). These computational models investigate the impact of different STDP timings or of modulators on STDP by integrating their effects on calcium dynamics or downstream signaling pathways. Therefore computational models based on the kinetics of the implicated molecular pathways are promising avenues to integrate the third factor in Hebbian plasticity and will be the main focus of the present review.

#### NEUROMODULATORS AFFECTING THE EXPRESSION, POLARITY AND SHAPE OF STDP

Neuromodulators and neurotransmitters play an important, but often unappreciated, role in the control of STDP induction and maintenance (for reviews see Pawlak et al., 2010; Edelmann et al., 2017). The skepticism about neuromodulation stems from the apparent discrepancy between the time scale of neuromodulation and the coincidence detection timing inherent to STDP. The former is on the scale of seconds or more, whereas the latter is on the scale of milliseconds. However, this apparent discrepancy becomes less important after considering STDP from the perspective of a learning system that needs to link recorded information (memory) with a value scale (reward). Indeed, an individual acting on its environment needs to learn to discriminate actions leading to reward from those leading to punishment, both possibly occurring seconds, minutes or even hours after the taken action. A system of memory and learning based only on the timescale of STDP would miss this essential information. Thus, one role of neuromodulation is to link STDP and the reward system. In this context, we demonstrate below how a third factor, comprised of neuromodulators and/or astrocytes, modulates the timing dependence of STDP. Note that the modulation of timing dependence depends on brain region and cell type; thus future computational models will need to incorporate region and cell type specific modulation. In this section, we detail STDP protocols used in experimental studies because depending on the activity patterns neuromodulatory systems are differentially recruited. Therefore, the apparent contradiction between several of the experimental reports on STDP could depend on the activity patterns or neuromodulatory activation that were used. This knowledge might help the building of computational models, by taking into account the different regimes of action of neuromodulators in shaping STDP.

### Dopamine

The action of dopamine is mediated by the metabotropic dopaminergic receptors that functionally modulate other receptor systems and/or ion channels without inducing large postsynaptic currents. Dopaminergic receptors belong to two groups based on their G-protein coupling: the D1-class receptors (D1R and D5R) are coupled to Gs- or Golf-proteins and the D2-class receptors (D2R, D3R and D4R) to Gi/o-proteins (Neve et al., 2004). D1- and D2-class receptors have opposite action on the cyclic adenosine monophosphate (cAMP) second messenger pathway and the protein kinase A (PKA; **Figure 1A**).

Dopamine is released by midbrain dopaminergic neurons in response to both reward and the reward prediction error (Schultz, 2007). In the hippocampus, tLTD, which is observed in control conditions for negative ∆tSTDP, is converted to tLTP by dopamine addition during STDP pairings or immediately after STDP pairings (aiming at mimicking a retroactive effect; Zhang et al., 2009; Brzosko et al., 2015; **Figure 1B**). Dopamine addition during STDP induction leads to the enlargement of the temporal window of tLTP expression (**Figure 1B**). However the effects of dopamine disappear when dopamine is added long after STDP pairings, since dopamine addition 10 and 30 min after pairings results in an absence of plasticity and a recovery of tLTD observed in control conditions, respectively (Brzosko et al., 2015; **Figure 1B**). This dopaminergic modulation, which converts bidirectional STDP to unidirectional tLTP, is D1R- but not D2Rmediated (Zhang et al., 2009; Brzosko et al., 2015). Acetylcholine (classically associated with arousal and exploratory behavior; Ma et al., 2018) transforms bidirectional Hebbian hippocampal STDP into unidirectional tLTD (Brzosko et al., 2017). However, the effect of acetylcholine is reverted by dopamine addition 1 s after STDP pairings, which allows recovering tLTP (**Figure 1B**). Although these results constitute an important step for the experimental demonstration of a retroactive action of dopamine

no permission required.

on Hebbian plasticity, the molecular mechanisms underlying dopamine interactions with the coincidence detectors were not characterized. In addition, more distal action of dopamine from STDP protocol remains to be investigated to fully explore the temporal credit-assignment problem (Sutton and Barto, 1998; Izhikevich, 2007; Schultz, 2007; Gerstner et al., 2018).

Additional evidence supports the role of dopamine for promoting hippocampal tLTP. Conditions that lower basal dopamine during the preparation of brain slices prevent the induction of tLTP at synapses between Shaffer collaterals and CA1 pyramidal cells (Edelmann and Lessmann, 2011). Subsequent addition of dopamine rescues tLTP, through a D1Rmediated mechanism (Edelmann and Lessmann, 2011, 2013). In addition, D1- and D5R-activations are important for the induction of tLTP at the synapses between the medial perforant pathway and dentate gyrus neurons (Yang and Dani, 2014). The mechanism here includes a change in cell excitability: inactivation of the transient A-type potassium current by D1R and D5R increases the excitability of dentate gyrus neurons and the amplitude of their back-propagating action potentials (Yang and Dani, 2014).

Beyond the hippocampus, the importance of dopamine modulation of STDP also is attested in the basal ganglia, where dopamine plays a crucial role in motor control, action selection and reinforcement learning (Yin and Knowlton, 2006; Schultz, 2007). Given the importance of dopamine, it is not surprising that dopamine is required for STDP in the striatum, both ex vivo (Pawlak and Kerr, 2008; Shen et al., 2008) and in vivo (Schulz et al., 2010; Fisher et al., 2017). However, the situation is complicated by the diversity in dopamine receptors. In rodents, striatal output neurons belong either to the direct or the indirect trans-striatal pathways and show different dopaminergic receptor expression, D1- and D2-class receptors, respectively (Calabresi et al., 2014). In vivo in anesthetized rodents, negative and positive pairing STDP protocol both result in tLTD at corticostriatal synapses, and bidirectional STDP can be elicited only with phasic dopaminergic release obtained by electrical stimulation of midbrain dopaminergic neurons (Schulz et al., 2010) or pharmacological manipulation of dopaminergic transmission (together with GABAergic and adenosine transmissions; Fisher et al., 2017). These results are consistent with ex vivo studies, which showed that application of dopamine either simultaneously, or 0.6 s after glutamate allows dendritic spine enlargement and calcium increase (Yagishita et al., 2014). Moreover, this study demonstrated the existence of synaptic eligibility traces, which can be revealed by subsequent dopamine release after Hebbian learning (see section ''Monoamines Transform Eligibility Traces Into Plasticity'' below). Ex vivo, conflicting results have been reported regarding STDP modulation by dopamine: according to Pawlak and Kerr (2008) both tLTD and tLTP requires D1R- but not D2R-activation (D2R-activation affecting only plasticity kinetics: tLTP and tLTD onset is shortened and delayed, respectively), whereas Shen et al. (2008) reported that D2R-activation is required for tLTD expression in striatal neurons belonging to the indirect pathway and D1R-activation is necessary for tLTP in striatal neurons belonging to the direct pathway. There are methodological differences between these two studies which could account for this discrepancy in results: for post-pre and pre-post pairings the same STDP protocol (i.e., 100 pairings at 0.1 Hz) was applied by Pawlak and Kerr (2008), whereas two distinct STDP-like protocols (theta bursts 3:3 for tLTP and 1:3 for tLTD) were utilized by Shen et al. (2008). Depending on the activity patterns, D1- and D2-class receptors could be differentially activated. The effects of dopamine in the striatum via D2R receptors would result from a D2R-mediated attenuation of both synaptic- and back-propagating action potential-evoked calcium influx into dendritic spines via the inhibition of PKA-dependent regulation of NMDARs (Higley and Sabatini, 2010; **Figure 1**). This mechanism also is supported by the demonstration that dopamine depletion enhances calcium influx in dendrites of the D2R-expressing striatal neurons belonging to the indirect pathway (Day et al., 2008). Future development of detailed computational models of the signaling pathways will be useful for fully exploring the involvement of dopaminergic receptors in various forms of STDP (see ''Molecular Pathway-Based Computational Models of STDP'' section).

The role of dopamine has been demonstrated in two other brain regions, the prefrontal cortex and the amygdala. In the prefrontal cortex (at layer 5 pyramidal cells) an STDP protocol such as 60 pairings (∆tSTDP = +10 ms) at 0.1 Hz fails to produce plasticity, while dopamine application during the STDP pairings permits the induction of Hebbian tLTP (∆tSTDP = +10 ms; Xu and Yao, 2010) and anti-Hebbian tLTP (∆tSTDP = −30 ms; Ruan et al., 2014; **Figure 1B**). Both Hebbian and anti-Hebbian tLTP directly depends upon D1Ractivation in the postsynaptic neuron whereas the Hebbian tLTP depends also indirectly upon the activation of D2R expressed by GABAergic interneurons. D2R activation blocks the inhibition exerted by GABAergic interneurons and permits the expression of Hebbian tLTP (∆tSTDP<+10 ms). By combining D1R- and D2R-activation, the temporal window of tLTP is extended up to ∆tSTDP = +30 ms (Xu and Yao, 2010; **Figure 1C**). This suggests that in prefrontal cortex, the physiological form of STDP is the anti-Hebbian tLTP since the expression of Hebbian tLTP is disfavored by GABAergic network activity. In the lateral nucleus of the amygdala tLTP requires the activation of D2R located on neighboring GABAergic interneurons (Bissière et al., 2003). Since dopamine is released in the amygdala in response to stress (Inglis and Moghaddam, 1999), dopaminergic neuromodulation of inhibitory synaptic transmission appears to be a crucial mechanism underlying the acquisition of fear conditioning.

In summary, these results show that dopamine is a key neuromodulator of STDP and constitutes the third factor required for the temporal credit-assignment. Overall, the effects of dopamine seem to conform to a simple general scheme: the activation of Gs/Golf-coupled D1R tends to promote tLTP whereas the activation of Gi-coupled D2R favors tLTD. However, the effects exerted by dopamine strongly depend on the brain area: dopamine either can be mandatory for STDP induction and/or maintenance or modulate STDP properties (width of the ∆tSTDP window, polarity of the STDP or magnitude of the plasticity). Moreover, network effects can add complexity to the picture, since the expression of dopamine receptors is not restricted to the examined neuron but can affect the response to e.g., local interneurons.

#### Noradrenaline

Noradrenaline interacts with G-protein–coupled receptors of three families: α2-, α1- and β1-3-adrenergic receptors (by order of decreasing affinity; Ramos and Arnsten, 2007). α2-adrenergic receptors are Gi/Go-coupled and lead to cAMP decrease. α1-adrenergic receptors are Gq-coupled, and activate phospholipase Cβ (PLCβ), resulting in intracellular calcium release via inositol 1,4,5-triphosphate (IP3). β-adrenergic receptors are Gs-coupled and yield cAMP increase (**Figure 2A**).

In several brain regions, activation of adrenergic receptors modifies the shape of the STDP curve (**Figure 2B**). In the hippocampus, activation of β-adrenergic receptors enlarges the range of ∆tSTDP for Hebbian tLTP expression by increasing the excitability of CA1 pyramidal cells (Lin et al., 2003). In the visual cortex, whereas paired stimulations of layer 4 afferents with postsynaptic action potential bursts does not produce plasticity, the concomitant activation of adrenergic receptors (with both α1-and β-adrenergic receptor agonists) allows the emergence of bidirectional Hebbian STDP in pyramidal cells of layer 2/3 (in rodents: Guo et al., 2012; in primates: Huang et al., 2014; **Figure 2B**) as well as in fast-spiking interneurons and non-fast-spiking somatostatin-positive interneurons (Huang et al., 2013). Note that α1-and β-adrenergic receptor agonists alone (or agonists of M1-class muscarinic acetylcholine receptors (mAChRs), see section below; Seol et al., 2007) trigger a tLTD-only (i.e., unidirectional anti-Hebbian STDP), whereas β-adrenergic receptor agonists alone induce the expression of a tLTP-only (i.e., unidirectional Hebbian STDP;

Seol et al., 2007; Guo et al., 2012; Huang et al., 2013; **Figure 2B**). The affinity for noradrenaline of α1-adrenergic receptors exceeds that of β-adrenergic receptors, and unidirectional anti-hebbian STDP (tLTD-only) is observed in low noradrenaline, whereas bidirectional Hebbian STDP can be induced with higher noradrenaline concentration (Salgado et al., 2012; **Figures 2B,C**).

Taken together, those studies show that adrenergic receptors play an important role in shaping STDP, mostly by enlarging ∆tSTDP and controlling STDP polarity, but also, similarly to dopamine, by acting subsequently to the stimulation to promote plasticity. Overall, a pattern emerges from the effects of noradrenaline: the activation of Gs-coupled β-adrenergic receptors tends to promote tLTP, whereas the activation of Gq-coupled α1-adrenergic receptors tends to favor tLTD.

### Monoamines Transform Eligibility Traces Into Plasticity

One of the fundamental questions in reward learning is the temporal credit-assignment problem: how are the correct actions learned given that delivery of a reward or punishment occurs significantly later than the key actions that promoted the outcome (Schultz, 2007). In an attempt to solve the temporal credit-assignment problem, some computational studies addressed the question of the retroactive effect of dopamine on cortical and hippocampal STDP (Sutton and Barto, 1998; Izhikevich, 2007; Gerstner et al., 2018). From a cellular perspective, the temporal credit-assignment problem translates into the following question: if dopamine (and more broadly monoamines) modulates STDP, is there a dependence of this modulation on the time elapsed between the stimulus (STDP pairings) and the reward (release of monoamines)? This question adds a supplementary temporal dimension to the modulation by the third-factor monoamine.

To solve the temporal credit-assignment or the distal reward problem, it has been proposed that synaptic eligibility traces could constitutes synaptic tags that are set by Hebbian learning and that will be transformed subsequently into synaptic plasticity by neuromodulators, bridging the learning sequence with reward (Sutton and Barto, 1998; Izhikevich, 2007; Gerstner et al., 2018). In other words, eligibility traces would be induced by Hebbian learning but would remain silent in term of synaptic efficacy changes, unless a neuromodulator released subsequently transforms them for plasticity. Synaptic eligibility traces would allow the synapse to keep a trace from the stimulus until getting the reward, the latter of which is represented by monoamines. We can distinguish two cases: the subsequent release of neuromodulator shapes an existing plasticity (Cassenaer and Laurent, 2012; Brzosko et al., 2015, 2017; Shindou et al., 2018) or allows the plasticity expression (Yagishita et al., 2014; He et al., 2015).

Octopamine, the equivalent of noradrenaline in insects, changes the bidirectional Hebbian STDP at synapses of Kenyon cells in the locust, critical for the associative learning of odors, into a unidirectional STDP (tLTD-only) even in a retroactive manner when applied seconds after the relevant pairing (Cassenaer and Laurent, 2012). In a similar way, in rodents, when dopamine is applied just after STDP pairings, it converts tLTD into tLTP in hippocampus (Brzosko et al., 2015, 2017) or in striatum (Shindou et al., 2018).

In striatum, dopamine induces spine enlargement exclusively when opto-stimulation of dopaminergic terminals occur between 0.3–2 s after Hebbian learning (i.e., STDP pairings; Yagishita et al., 2014). In the visual cortex and in the medial prefrontal cortex, release of noradrenaline and serotonin, just after the whole set of pairings or just after every pairing, allows the expression of tLTP and tLTD for pre-post and post-pre pairings, respectively (He et al., 2015); the STDP pairings per se did not induce plasticity (**Figure 2C**). He et al. (2015) observed that the eligibility traces are short-lived since the monoamines need to be release 5–10 s after learning to promote plasticity (He et al., 2015). The fact that a couple of monoamines (or third factors) is at play for distinct induction plasticity (tLTP vs. tLTD) could allow an efficient stabilization of learning and avoid synaptic saturation.

#### Acetylcholine

Acetylcholine acts on two types of muscarinic receptors mAChRs: the M1-(M1, M<sup>3</sup> and M5) and M2-(M<sup>2</sup> and M4) class receptors (Thiele, 2013), and the ionotropic (cationic) nicotinic acetylcholine receptors (nAChRs; Albuquerque et al., 2009). M1-class mAChRs are Gq/G11-coupled leading to IP3 and diacylglycerol (DAG) production (via PLCβ activation), subsequent increase of intracellular calcium and activation of protein kinase C; (PKC; **Figure 2A**); M2-class mAChRs are Gi/Go-coupled, leading to inhibition of adenylate cyclase, and a reduction of cAMP and thus PKA activity.

Unlike STDP experiments with noradrenaline and dopamine, experiments to characterize the effect of acetylcholine have not carefully delineated M1-class vs. M2-class effects; thus experimental results are more diverse. At hippocampal CA1 pyramidal cells, bidirectional Hebbian STDP is converted into unidirectional tLTP after enhancement of acetylcholine (Brzosko et al., 2017; **Figure 1B**), whereas inhibition of mAChRs prevents post-pre tLTD and converts pre-post tLTP into tLTD (Sugisaki et al., 2011, 2016). When excitatory and inhibitory post-synaptic currents were examined at synapses of CA1 pyramidal neurons, pre-post pairings induce tLTP of excitatory pathway while it triggers tLTD at inhibitory pathways via the co-activation of mAChRs and CB1R (Ahumada et al., 2013). Thus, Hebbian STDP in CA1 pyramidal neurons depends on the excitation/inhibition balance, which is tightly regulated by mAChRs expressed in GABAergic interneurons and pyramidal cells (Sugisaki et al., 2011, 2016; Ahumada et al., 2013; Takkala and Woodin, 2013).

Though acetylcholine alone seems to promote unidirectional plasticity (tLTP- or tLTD-only), co-activation of mAChRs and Gs coupled pathways (either D1/D5 dopaminergic receptors in the hippocampus CA1 pyramidal cells (Brzosko et al., 2017) or β-adrenergic receptors in visual cortex layer 2/3 pyramidal cells (Seol et al., 2007) promotes bidirectional plasticity by restoring Hebbian tLTP for ∆tSTDP > 0 (**Figure 2B**).

The effects of acetylcholine via nAChR-activation are expected to include depolarization and possibly increased calcium influx (Jones et al., 2012), but they also can exert a more subtle influence on STDP by regulating the magnitude of STDP rather than its polarity or expression (Sugisaki et al., 2016). Nicotine increases the threshold for the induction of Hebbian tLTP at excitatory synapses of pyramidal cells of the prefrontal cortex (Couey et al., 2007). However, note that nicotine when applied at a high concentration (∼10 µM) can exert a more drastic effect on STDP since it converts tLTP into tLTD (Couey et al., 2007). Interestingly, in the medial prefrontal cortex, after nicotine treatment in juvenile rats, opposing effects are obtained depending on the developmental stage: tLTP magnitude was reduced in juvenile whereas it was increased in adult rats (Goriounova and Mansvelder, 2012).

Taken together, the above results reveal a general principle whereby the neuromodulatory effects exerted on STDP by monoamines (dopamine or noradrenaline) or acetylcholine are for a large part guided by the type of G-protein activated (regardless of the agonist): Gi/o-coupled and Gq/11-coupled receptor activation facilitates tLTD (D2-class, α1-adrenergic, M1-class), whereas Gs- and Golf-coupled receptor activation rather leads to the expression of tLTP (D1R-class, β-adrenergic receptors). However the validity of this general principle needs further investigation in other brain areas and neuronal subtypes.

### Brain-Derived Neurotrophic Factor (BDNF)

The neurotrophic factor brain-derived neurotrophic factor (BDNF) binds to the tyrosine receptor kinase B, which induces tyrosine receptor kinase B dimerization and the autophosphorylation of tyrosine residues in the cytoplasmic kinase domain. This process induces the activation of three main signaling pathways: phospholipase Cγ, phosphoinositide 3-kinase and extracellular signal-regulated protein kinases cascades. Notably, the phosphoinositide 3-kinase signaling pathway plays an important role in the regulation of mRNA translation, which impacts protein synthesis and putatively BDNF-dependent plasticity. Numerous studies have shown the role of BDNF in modulating synaptic transmission and plasticity (for reviews see Park and Poo, 2013; Edelmann et al., 2014).

Concerning STDP, pairings of glutamate release and postsynaptic spiking at 1 Hz are sufficient to release BDNF from the postsynaptic dendrites in a spike-timing-dependent manner (for 0 < ∆tSTDP ≤ +20 ms; for ∆tSTDP > 20 ms BDNF release was not detected; Lu et al., 2014). This spiketiming-dependent BDNF release is dependent on the activation of NMDARs. In hippocampal neurons, the tLTP part of the observed bidirectional Hebbian STDP depends on BDNF (Bi and Poo, 1998; Lu et al., 2014). Interestingly, depending of the activity pattern during STDP pairings, the BDNF dependence of the observed plasticity is different. Indeed, hippocampal tLTP induced with presynaptic activation paired with postsynaptic bursts of four back-propagating action potentials (1:4 pairings repeated 30 times at 0.5 Hz) is BDNF and tyrosine receptor kinase B-mediated, whereas canonical STDP pairings (1:1 pairings repeated 100 times at 0.5 Hz) induced a tyrosine receptor kinase B-independent tLTP at the same synapses (Edelmann et al., 2015). Genetic impairment of BDNF synthesis has led to alteration of STDP in the prefrontal cortex. Disruption of one of the promoters involved in BDNF transcription (promoter IV mutant mice) leads to the aberrant induction of tLTP, which is absent in wild-type mice, for 50 pairings (Sakata et al., 2009). In the infralimbic medial prefrontal cortex, STDP is absent in a rodent model (BDNF-Met/Met mice) of the human BDNF Val66Met polymorphism (leading to severe cognitive dysfunction and anxiety disorders) in which the BDNF release is impacted; STDP is recovered after exogenous BDNF application (Pattwell et al., 2012).

#### Nitric Oxide (NO)

Nitric oxide (NO), an intercellular messenger, is generated by the enzyme NO synthase and activates soluble guanylyl cyclase leading to cyclic guanosine monophosphate (cGMP) formation. In turn, cGMP-activated protein kinases regulate multiple substrates such as DARPP-32 and G-substrate, which inhibits phosphatases that are involved, among other effects, in synaptic plasticity expression (for review see: Hardingham et al., 2013). Concerning STDP, in the somatosensory cortex of mice, Hebbian tLTP depends both on the α-amino-3-hydroxy-5 methyl-4-isoxazolepropionic acid receptor (AMPAR)-subunit-1 and a NO-dependent presynaptic component (Hardingham and Fox, 2006). Similarly, glutamate afferents to serotoninergic neurons of the dorsal raphe nucleus exhibit tLTP for pre-post pairings, which is NO-dependent, involving the cGMP-activated protein kinase signaling cascade (Haj-Dahmane et al., 2017). In retinal ganglion cells of tadpoles, STDP can be induced by natural visual stimulation (e.g., moving bar) or by electrical stimulation of the retina and in both cases, NO is required for tLTD while BDNF is required for tLTP (Mu and Poo, 2006).

### GABA

In the dorsal striatum, anti-Hebbian STDP as observed in control conditions in striatal output neurons is shifted to Hebbian STDP under pharmacological blockade of GABAAR receptors (Fino et al., 2010; Paille et al., 2013; Valtcheva et al., 2017; **Figure 3A**). This effect applies equally at D1R-class striatopallidal (direct pathway) and D2R-class striatonigral (indirect pathway)

neurons of juvenile and adult rodents. Although the molecular mechanisms underneath this reversal of polarity by GABA are not fully elucidated, a computational model suggests that depolarizing effects of GABA at distal dendrites would reverse calcium influx by modifying the balance between calcium influxes from NMDAR vs. voltage-sensitive calcium channels (Paille et al., 2013). Although GABA increases calcium influxes in both NMDAR and voltage-sensitive calcium channels, via its depolarizing effect in striatal output neurons (due to the relative values of the chloride reversal and membrane potential), the depolarizing effect of GABA would impact differentially Foncelle et al. Modulation of STDP

NMDAR and voltage-sensitive calcium channels depending on the order of pairings (post-pre vs. pre-post). GABA would favor calcium influx via voltage-sensitive calcium channels for post-pre pairings (promoting tLTP), whereas it would favor calcium influx via NMDARs for pre-post pairings (promoting tLTD) in control conditions, leading to anti-Hebbian STDP (Paille et al., 2013). Under GABA blockade, this balance between calcium influxes is shifted and Hebbian STDP can be observed. Change in GABAergic signaling during striatal development (i.e., the onset of the tonic GABAergic signaling around P14; Ade et al., 2008) appears to be a key factor for shaping of striatal STDP. Indeed, in young rats (P7–10) corticostriatal STDP is unidirectional and Hebbian (tLTD with post-pre pairings, no plasticity with pre-post pairings) but bidirectional and anti-Hebbian in adult rodents (Valtcheva et al., 2017; **Figure 3A**). GABA signaling is also at play with the control of CB1R-dependent tLTP which expression shifts from post-pre to pre-post stimulation when ionotropic GABA<sup>A</sup> transmission is blocked (Cui et al., 2015). GABA is not involved in the induction of STDP per se, nor its magnitude, but controls STDP polarity, i.e., the association between the sign of the pairing (prepost or post-pre) and the plasticity outcome (tLTP or tLTD). The tonic GABAergic component plays a major role in the emergence of the anti-Hebbian striatal STDP in juvenile and adult rodents (Valtcheva et al., 2017; **Figure 3A**). Thus, the pathological deregulation of tonic GABAergic signaling may affect the polarity and occurrence of striatal plasticity and alter procedural learning and memory. It remains to be seen whether the neuromodulator role of GABA for STDP emergence and/or polarity constitutes a general rule in the brain.

Change of STDP polarity induced by GABAergic transmission has also been observed in hippocampus. In hippocampal CA1 pyramidal cells, blockade of GABAARs converts unidirectional tLTD to bidirectional Hebbian STDP (with 80 pairings at 5 Hz; Sugisaki et al., 2016; **Figure 3B**). The modulatory effects of GABA<sup>A</sup> and GABA<sup>B</sup> receptors can also combine. Indeed, at Schaffer collateral-CA1 excitatory synapses of the rat hippocampus, plasticity relies on postsynaptic GABA<sup>A</sup> receptors to set the spike-timing dependency and also depends on presynaptic GABA<sup>B</sup> receptors for its frequency dependence (Nishiyama et al., 2010; **Figure 3B**). Specifically, postsynaptic GABA<sup>A</sup> receptors regulate the timing dependence of tLTD at 5 Hz pairings (in the theta frequency band), whereas presynaptic GABA<sup>B</sup> receptors control the frequency dependence of tLTD at 25 Hz (alpha and beta frequencies) and also accounts for the expression of tLTP for 5 Hz and 50 Hz (gamma frequencies; Nishiyama et al., 2010). In addition, STDP can be expressed at GABAergic interneurons, where it modulates the strength of GABAergic inhibition since STDP pairings alters the activity of potassium-chloride cotransporter-2, resulting in changes in the reversal potential of GABAergic synaptic currents (Woodin et al., 2003).

Taken together, the above results indicate that the spectrum of the third factor of STDP is very large since in addition to neuromodulators it can be extended to BDNF, NO and neurotransmitters acting as neuromodulators such as GABA. STDP synaptic plasticity is thus modulated, whether in its induction, its direction or its temporal window. Though neuromodulation of STDP has been investigated for the early phase of plasticity (within the first hour, i.e., the induction phase), the effects of neuromodulators remain to be investigated for the late phases of plasticity in which the third factor is expected to have a crucial role for the maintenance of memory (Lisman et al., 2011).

# Modulation of STDP by Astrocytes: the Forgotten Third Factor

Many forms of excitatory STDP rely on either pre- or postsynaptic glutamate receptors (Sjöström et al., 2008; Feldman, 2012; Korte and Schmitz, 2016). Therefore, STDP is expected to be tightly controlled by glutamate dynamics. Specifically, the spatiotemporal profile of glutamate may define the extent and location of recruited glutamate receptors, which are involved in the induction of tLTP or tLTD.

An overriding question is how coincident synaptic activity in the millisecond range can be integrated over a longer timescale during the iteration of pre- and postsynaptic pairings to allow STDP induction, while keeping sharp sensitivity to timing during individual pairing episodes. A potential solution to this problem could be that: (1) glutamate should be released in a delayed manner to allow integration of pre- and postsynaptic activity over the time course of minutes; and (2) synaptically released glutamate during neuronal activity needs to be reliably cleared from the extracellular space to allow high fidelity sampling of coincident pre- and postsynaptic activity during STDP pairings. Astrocytes help solve this problem of controlling extracellular glutamate dynamics and have been shown to play an important role in synaptic transmission, as well as short- and long-term memory (Chung et al., 2015; Oliveira et al., 2015). This role of astrocytes has led to the concept of the tripartite synapse, comprised of the pre- and postsynaptic neuronal elements as well as the astrocytes. Indeed, a substantial part of central synapses are contacted by astrocytes (Bernardinelli et al., 2014). Notably, astrocytes are able to release glutamate via exocytosis in response to neuronal activity (Araque et al., 2014; Sahlender et al., 2014; Verkhratsky et al., 2016) and to efficiently clear glutamate from the extracellular space on a submillisecond timescale via high-affinity glutamate transporters (Danbolt, 2001). Therefore, astrocytes can both detect and control neuronal activity via the release and reuptake of glutamate.

Astrocytes can integrate the coincident neuronal activity during STDP pairings and participate in the induction of tLTD (Min and Nevian, 2012). Excitatory tLTD induced by post-pre pairings at layer 4 onto layer 2/3 synapses in the rat barrel cortex relies on the release of endocannabinoids by the postsynaptic element through the activation of astrocytic CB1Rs (Min and Nevian, 2012). In turn, glutamate released by astrocytes activates presynaptic NMDARs which are required for tLTD induction (Rodríguez-Moreno and Paulsen, 2008). Astrocytes are able to sense postsynaptic endocannabinoid release by gradually increasing their calcium waves exclusively during repetitive post-pre pairings within a narrow temporal window of ∆tSTDP = −25 ms which is eligible for tLTD induction. Indeed, pre-post pairings at ∆tSTDP = +25 ms and post-pre pairings at ∆tSTDP = −250 ms, which induce tLTP and no plasticity, respectively, do not trigger any changes in calcium dynamics (Min and Nevian, 2012). Therefore, astrocytes are selective to a unique temporal pattern, which both generates calcium dynamics to promote glutamate release and imposes a threshold for tLTD induction. Astrocytes can thus act as a time buffer by integrating coincident pre- and postsynaptic activity over the time course of minutes and enabling tLTD by delayed release of glutamate.

Astrocytes also are crucial for the gating of both tLTP and tLTD in the dorsal striatum via the uptake of glutamate (Valtcheva and Venance, 2016). Physiological activity of the astrocytic glutamate transporter, called the excitatory amino acid transporter-2 (EAAT2), allows the expression of bidirectional anti-Hebbian STDP induced in a narrow temporal window −30 < ∆tSTDP < +30 ms (Fino et al., 2010; Paille et al., 2013; Valtcheva and Venance, 2016). When EAAT2 is blocked, a form of LTP that does not rely on coincident detection can be induced by uncorrelated activation of pre- and postsynaptic elements. This non-Hebbian LTP requires postsynaptic back-propagating action potentials and extrasynaptic GluN2B-containing NMDARs, which are activated by glutamate spillover. In contrast, the overexpression of EAAT2 prevents the expression of striatal STDP (Valtcheva and Venance, 2016) possibly by restricting glutamate availability for both the NMDARs and mGluRs required for striatal STDP (Shen et al., 2008; Fino et al., 2010). Thus, preserving the optimal temporal contingency between pre- and postsynaptic activity required for STDP depends on astrocytic glutamate uptake. Astrocytes gate tLTP and tLTD by a subtle regulation of the extracellular glutamate levels and, therefore, a precisely tuned range of EAAT2 activity allows the emergence of STDP. Computational models have begun to explore interactions between glutamatergic synapses and astrocytes (De Pittà et al., 2011; De Pittà and Brunel, 2016 see also De Pittà et al. (2012) for a review), but investigating the role of astrocytic glutamate control requires transforming the binary glutamate release event typically used in STDP models into glutamate diffusion and update mechanisms.

Astrocytes can release various other neurotransmitters and factors besides glutamate (Araque et al., 2014; Sahlender et al., 2014; Verkhratsky et al., 2016) including the NMDAR co-agonist D-serine which regulates different forms of synaptic plasticity. The release of D-serine is necessary for frequency-dependent LTD and LTP in the hippocampus (Zhang et al., 2008; Henneberger et al., 2010) and prefrontal cortex (Fossat et al., 2011). Moreover, experience-dependent changes in the degree of synaptic enwrapment by astrocytes governs the level of D-serine availability and subsequently controls the expression of NMDAR-dependent LTP and LTD in the supraoptic nucleus of the hypothalamus of lactating rats (Panatier et al., 2006). The NMDARs implicated in STDP can be situated at both pre- or postsynaptic sites (Feldman, 2012; Korte and Schmitz, 2016) and thus may be affected to different extents by gliotransmission. D-serine has a permissive role for the induction of NMDAR-dependent tLTP at mossy fiber-CA1 hippocampal synapses (Rebola et al., 2011), although its glial origin has not been investigated. In the developing hippocampus a presynaptic tLTD at CA3-CA1 synapses requires D-serine signaling possibly released from astrocytes (Andrade-Talavera et al., 2016). Interestingly, the same STDP pairing protocol induces tLTP at later developmental stages suggesting the possibility that astrocytic coverage of neurons and modulation of STDP by gliotransmission may be developmentally regulated.

Another important gliotransmitter is adenosine triphosphate (ATP), which is enzymatically converted to adenosine in the extracellular space and can act on pre- and postsynaptic adenosine receptors situated on neurons. Glial release of ATP controls the magnitude of hippocampal LTP induced with high-frequency stimulation (Pascual et al., 2005) and blockade of postsynaptic adenosine A2a receptor increases the amplitude of low-frequency stimulation-dependent LTD in the striatum (Lerner et al., 2010). Adenosine also mediates striatal tLTP via postsynaptic adenosine A2a receptors both in vitro (Shen et al., 2008) and in vivo when the STDP paradigm is coupled with dopamine pairing (Fisher et al., 2017). In addition, presynaptic adenosine A1 receptors modulate the amplitude of tLTP in the visual cortex (Bannon et al., 2017). However, evidence directly implicating astrocytes in the purinergic control of STDP is still lacking. Computational models of signaling pathways underlying STDP have begun to include adenosine A2a receptors (see below), but investigation of interaction between pre-synaptic NMDA and adenosine A1 receptors requires modeling of mechanisms controlling pre-synaptic vesicle release.

Finally, astrocytes are involved in the GABAergic modulation of both the polarity (Fino et al., 2010; Paille et al., 2013; Valtcheva et al., 2017) and threshold for induction (Groen et al., 2014) of excitatory STDP. Astrocytes regulate basal and transient inhibitory tone via GABAergic transporters (Scimemi, 2014). Non-specific blockade of both neuronal and astrocytic GABA transporters in the developing striatum has a permissive role for the induction of tLTD (Valtcheva et al., 2017) but the particular contribution of astrocytic GABA clearance in STDP remains to be explored.

#### MOLECULAR PATHWAY-BASED COMPUTATIONAL MODELS OF STDP

In an attempt to better understand the mechanisms governing learning and memory and determine which mechanisms control input-dependent plasticity, modeling efforts have focused on biophysical and biochemical models that utilize a kinetic description of the molecular pathways implicated in STDP. These models range in molecular complexity from single ion (i.e., calcium) to complicated signaling pathways, and in spatial complexity from single-compartment (**Figure 4A1**) to multicompartment (**Figure 4A2**). An overview of this literature can be found in several review articles (see e.g., Graupner and Brunel, 2010; Griffith et al., 2015). In the following, we focus on the articles published after 2010, though include the most influential contributions published before that date. Moreover, in the following, we subdivide the models into two types: those evaluating the control of plasticity from calcium dynamics alone, and those that add one or more downstream signaling pathway

FIGURE 4 | Computational models for predicting the direction of STDP have a wide range of complexity. (A) Models differ in morphological complexity, from single-compartment (A1) to multi-compartment models (A2). Top traces show that the back-propagating action potential decreases in amplitude, initiates later and broadens as it propagates distally in multi-compartmental models. Bottom traces show that distal synapses may produce higher calcium elevations than proximal synapses due to higher local input resistance. (B) Models differ in the mechanisms used to control calcium dynamics, from single time constant of decay, to biophysical/biochemical models of diffusion (red arrows), pumps (such as the plasma membrane ATPase: PMCA) that extrude calcium (yellow arrow), buffers (such as calmodulin, calbindin, or immobile buffers) that bind to free calcium (gray arrows) and calcium release (not shown). All models include influx through the N-methyl-D-aspartate receptors (NMDARs; blue arrows). (C) The prediction of plasticity from calcium often uses two amplitude thresholds (C1), but sometimes include duration thresholds (C2) or other measures of calcium duration. TLTP, tLTP amplitude threshold; TLTD, tLTD amplitude threshold; DLTD, threshold on the duration of the calcium elevation.

molecules. In addition, we try to distinguish single-compartment models from those that add some degree of spatial structure to the postsynaptic neuron. We acknowledge that in both of these dimensions the classification is not binary and some models bridge the divide.

# Simplified Calcium Dynamics and Two-Threshold Rules

Models of calcium dynamics in response to STDP stimuli are the most common type of models, and are justified both by the critical role of calcium in plasticity and also by the stimulation protocol in which neuromodulator release does not change. The only difference between STDP protocols that produce tLTP and STDP protocols that produce tLTD is the timing between the presynaptic stimulation and the postsynaptic action potential, ∆tSTDP; thus the number and frequency of presynaptic stimulations does not differ between tLTP and tLTD. This implies that presynaptic release of neuromodulators does not differ so it must be postsynaptic molecules activated by calcium dynamics that determine the polarity of plasticity.

Calcium predicting the direction of synaptic plasticity is one of the ideas that are popular among theoreticians and experimentalists. In the simplest form the peak calcium (or indeed the amplitude of the current through the calcium permeable, NMDA subtype of the glutamate receptor) controls the direction of plasticity (for reviews see: Graupner and Brunel, 2010; Evans and Blackwell, 2015; Griffith et al., 2015). This is known as the ''two-threshold'' rule: if calcium (either peak or integrated) is above the higher, potentiation threshold, tLTP is induced, whereas if calcium is larger than the lower LTD threshold but lower than the LTP threshold, tLTD occurs (**Figure 4C1**). Pre-post pairings produce a large calcium influx through the NMDA receptor channel with calcium concentration above the LTP threshold, whereas post-pre pairings produce a moderate calcium influx with calcium concentration between the LTD and LTP thresholds. One of the first models of NMDAR-dependent synaptic plasticity was proposed by Shouval et al. (2002). This model, using simplified calcium dynamics inside a dendritic spine, accounted for a diverse range of stimulation protocols such as STDP and classical rate-based plasticity; however it predicted depression for long positive ∆tSTDP, a model prediction which is not confirmed by experiments (but see Nishiyama et al., 2000, 2010; Wittenberg and Wang, 2006). In the dorsal striatum, a model of calcium dynamics (Evans et al., 2012) evaluated the role of NMDAR subunit (2A and 2B subunits) in shaping the sensitivity to timing dependence, and correctly predicted that NMDAR-2A would require a small ∆tSTDP, whereas NMDAR-2B can support tLTP with a large ∆tSTDP. Several extensions or modifications to the basic model have been made both to account for results with spike triplets (i.e., when either two presynaptic stimuli or two postsynaptic action potentials are generated) and to minimize the tLTD window for long positive spike-timings. Adding another coincidence detection of presynaptic NMDARs with endocannabinoids is one mechanism utilized in a neuromorphic implementation of calcium based synaptic plasticity (Rachmuth et al., 2011). Alternatively, incorporating short term depression of transmitter release or AP back-propagation (Shouval and Kalantzis, 2005; Bush and Jin, 2012) minimizes the tLTD seen with long positive ∆tSTDP and can account for other experimental results; however, a more broadly applicable study (Rubin et al., 2005) showed that plasticity rules that use calcium amplitude alone cannot completely avoid predicting tLTD for long positive timings.

An extension of the two-threshold rule states that the duration of calcium elevation is equally important in determining direction of plasticity (**Figure 4C2**). Several models of STDP explicitly take into account both the amplitude and the duration of calcium in predicting plasticity outcome (Kumar and Mehta, 2011; Graupner and Brunel, 2012). Including a duration threshold or integrating the total calcium response allows correctly predicting experimental outcomes for both traditional STDP curves and STDP curves produced by spike triplets. Another extension of the Shouval et al. (2002) model, Standage et al. (2014), implements a calcium-dependent, sigmoid-shaped time constant of calcium decay, which represents saturation of calcium extrusion from the spines. This model shows that saturation of calcium extrusion might be responsible for the dependence of tLTP on the (theta-frequency like) inter-spike interval for triplet stimulation protocols. Including the duration of calcium does not exclude consideration of presynaptic release probability on STDP. Indeed, gliotransmission may change the shape of the STDP curve depending on whether gliotransmitters increase or decrease presynaptic release (De Pittà and Brunel, 2016).

# Threshold Rules Based on Detailed Calcium Dynamics

Most of the aforementioned models use simplified calcium dynamics instead of explicitly implementing the mechanisms underlying control of calcium (**Figure 4B**), which might improve predictions of synaptic plasticity. In other words, the next set of models used neither single time constant of decay nor summation of independent pre- and postsynaptic components for calcium dynamics. Though not explicitly implementing a STDP rule, Griffith et al. (2016) indirectly consider the effect of calcium duration by using calcium-bound calmodulin to assess how back-propagating action potential timing influences calcium concentration. Using a 3-dimensional, deterministic reactiondiffusion model of calcium interactions with calmodulin and other calcium binding proteins within a dendritic spine, Griffith et al. (2016) show that calcium-bound calmodulin is a more sensitive indicator of spike timing than free calcium. They further demonstrate the role of neuromodulators in regulating synaptic plasticity through their activation or inhibition of calcium dependent potassium channels during an STDP protocol, which greatly modulates calmodulin activation.

Several studies explicitly investigate how the dendritic location and inhibitory inputs shape the local calcium-based plasticity rules (Bar-Ilan et al., 2013; Je˛drzejewska-Szmek et al., 2016). Bar-Ilan et al. (2013) showed that inhibition shapes the spatial profile of dendritic calcium concentration in neocortical pyramidal neurons. Depending on the location of the excitatory and inhibitory inputs on the dendritic tree (**Figure 4A2**), tLTP may be blocked, transformed to tLTD, or the synapse may undergo no plasticity. Similarly, Je˛drzejewska-Szmek et al. (2016) developed a computational model of the major neuron type in the striatum, the striatal output neurons, including both electrical activity and calcium dynamics. They demonstrated that calcium amplitude and duration together (**Figure 4C2**) can predict a wide range of experimental plasticity outcomes, and further demonstrated a distance dependence of STDP caused by the back-propagating action potential. In both of these models, the distance dependent decreases in back-propagating action potential amplitude reduces calcium influx through NMDARs for more distant synapses. This reduced calcium influx can convert tLTP into either tLTD or no plasticity. These publications demonstrate that by modeling mechanisms controlling calcium dynamics, including diffusion, buffers and pumps, and by considering calcium duration, the LTD window for long positive ∆tSTDP is avoided.

An aspect of calcium dynamics often ignored in modeling studies is calcium release from intracellular stores. This has been shown to contribute to tLTP under some conditions (Plotkin et al., 2013; Cui et al., 2016). Thus, Nakano et al. (2013) included calcium release from stores in their multi-compartmental model of a direct pathway spiny projection neuron. In addition, though not explicitly including other signaling pathways, they evaluated the effect of dopaminergic modulation of calcium, potassium and NMDAR channels. The main result of their simulations showed that dopaminergic input preceding a back-propagating action potential induced higher calcium responses than dopamine input following a back-propagating action potential. This study also predicted that the timing dependence of calcium responses between the up- and down-states was similar.

# Models of Signaling Pathway to Explain Synaptic Plasticity

Beyond calcium, several models add on simplified or abstract version of downstream signaling molecules. Rubin et al., 2005 propose a three detector system, loosely based on pathways resembling the opposing Ca2+/calmodulin-dependent protein kinase-II (CaMKII)—protein phosphatase signaling pathways. In brief, three calcium-sensitive detectors are implemented: high, transient calcium levels activate the tLTP detector; low calcium elevations activate the tLTD detector; and intermediate calcium levels activate a ''Veto'' detector. Another variable integrates both the tLTD detector and the Veto detector (called a double filter), such that intermediate calcium levels decrease the double filter value; thus the double filter detects the uninterrupted duration of calcium at low values yet suppresses the development of tLTD should calcium spend some time at intermediate values, such as occurs with long positive ∆tSTDP. Using the three calcium detector system of Rubin et al. (2005), Cutsuridis (2011) showed that single GABAergic inhibitory inputs can sharpen the shape of the STDP curve: narrowing the temporal window that supports tLTD, whereas a train of GABAergic inputs both sharpens the tLTD window and reduces the tLTP amplitude. A follow-up study (Cutsuridis, 2012) extended the model to burst stimulation, and predicted that GABAergic inputs would expose a tLTD window for long positive ∆tSTDP. The timing of the GABA inputs determined whether the effect was predominantly depression or potentiation.

Several models (Graupner and Brunel, 2007, 2012; Pi and Lisman, 2008; Carlson and Giordano, 2011; Mihalas, 2011; Saudargiene. and Graham, 2015; Cui et al., 2016; for reviews see: Graupner and Brunel, 2010; Evans and Blackwell, 2015; Griffith et al., 2015) have implemented even more realistic representations of signaling pathway kinetics, including the calcium activated phosphatase calcineurin, the calcium activated kinase, CaMKII, and the Gs-activated adenylyl cyclase, the latter of which produces cAMP to activate PKA (**Figure 5**). Additional pathways, such as PKC (resulting from activation of Gq-coupled receptors such as M1R and mGluR) and extracellular signalregulated protein kinase (ERK; downstream of protein kinases A, C and tyrosine receptor kinase B) are also involved. Several advantages accrue from these models, including the ability to produce experimentally testable predictions regarding the role of specific molecules. Another key advantage of simulating signaling molecules is that the tLTD window for long positive ∆tSTDP is eliminated without arbitrarily assuming the existence of a dedicated calcium concentration range that does not elicit synaptic plasticity, i.e., a separate range between the tLTD-inducing calcium range and the tLTP range. Again these models vary in complexity, such as the number of different signaling pathways included, and whether spatial aspects are included. Several models of these signaling molecules have been applied to STDP protocols in the cortex, hippocampus and striatum.

One of the earliest models, the single-compartment electric model of Graupner and Brunel (2007), couples membrane potential with a biochemical reaction model via calcium dynamics. Phosphorylation state of CaMKII serves as the models readout, i.e., the level of phosphorylated CaMKII serves as a proxy of the synaptic weight. Short positive intervals can switch

when the spike timing and the number of pairing varies. tLTD progressively emerges at positive ∆tSTDP, whereas for negative ∆tSTDP, the model correctly predicts two domains of tLTP, one around 10–20 pairings and another emerging after 50 pairings. (C) When CB1R are blocked in the model, both the tLTD and the tLTP for low pairing numbers disappear. (D) Adding presynaptic D2Rs in the model, correctly predicts that tLTP for low pairing numbers is also controlled by dopamine. Adapted from Cui et al. (2016) with no permission required.

the CaMKII to a highly phosphorylated state; whereas negative intervals (but not long positive intervals) switch the CaMKII to a low phosphorylated state. Critical to success of this model is adjustment of calcium dependence of PKA and calcineurin activity against inhibitor 1, which controls the level of free protein phosphatase 1. A high-level of protein phosphatase 1 will dephosphorylate CaMKII to prevent its persistent activation. Indeed, in this model (**Figure 5**): (i) the protein phosphatase-1 (PP1)/CaMKII activation ratio dictates plasticity; LTD is expressed when PP1 activation overcomes CaMKII, whereas LTP occurs when CaMKII activation is larger than PP1 activation; and (ii) PP1 activity is maximal at intermediate calcium levels whereas CaMKII activation needs larger calcium levels. Short negative ∆tSTDP yield intermediate but long lasting calcium levels, which efficiently activate PP1 but are not large enough to activate CaMKII, thus triggering LTD. Short positive ∆tSTDP yield sharp calcium peaks that are large enough to activate CaMKII but do not persist long enough around intermediate values to activate PP1 significantly; this leads to LTP. Finally, the calcium levels triggered by long positive ∆tSTDP are too weak to activate CaMKII but do not stay long enough around intermediate values to activate PP1. Long positive ∆tSTDP therefore fail to activate either the PP1 or CaMKII, which in effect rules out the expression of tLTD. This molecular system therefore exhibits dynamics similar to the ''Veto'' detector proposed by Rubin et al. (2005) to eliminate tLTD at long positive spike timings (see above).

Subsequent models either enhance the electrical activity model, or add AMPA receptors as a model readout. Urakubo et al. (2008) develop a multi-compartment, multi-ion channel model of visual cortex pyramidal neurons to activate a biochemical reaction model. In contrast to Graupner and Brunel (2007), the timing dependence of tLTD cannot be reproduced unless calcium-bound calmodulin allosterically inhibits NMDARs. Both Carlson and Giordano (2011) and Saudargiene. and Graham (2015) used a model of AMPAR insertion controlled by the CaMKII/protein phosphatase-2A switch. Carlson and Giordano (2011) used a single-compartment model of calcium dynamics (from Shouval et al., 2002) to activate the biochemical network model of Pi and Lisman (2008). This single-compartment model can explain STDP and does not predict tLTD for long positive ∆tSTDP. Voltagesensitive calcium channels are critical for the latter effect, as blocking voltage-sensitive calcium channels allow tLTD to emerge for long positive ∆tSTDP. Saudargiene. and Graham (2015) incorporated spatial aspects of calcium dynamics by using a detailed compartmental model of pyramidal CA1 neuron (Poirazi et al., 2003) to activate a biochemical network model derived from two earlier models (Graupner and Brunel, 2007; Pi and Lisman, 2008). Saudargiene. and Graham (2015) showed, by monitoring AMPAR phosphorylation by the CaMKII/protein phosphatase-2A switch, that tLTD is indeed induced by lower calcium levels than tLTP, and that tLTD also requires many more repetitions of this lower calcium (which is consistent with experimental results). Saudargiene. and Graham (2015) also investigated the influence of particular timings of inhibition associated with excitatory inputs, showing that inhibition affects tLTD more that tLTP, because tLTD occurs for moderate calcium levels and is thus more vulnerable to any reduction in peak calcium.

Whereas spatial models of calcium dynamics typically include dendritic branching or explicit spines (microdomains), many signaling molecules are anchored via structural proteins into multi-protein complexes, effectively creating nanodomains of molecule interactions. One method for evaluating the effect of nanodomains (without explicitly creating a spatial model) is to couple different sources of calcium to different downstream signaling molecules. This approach was utilized by Mihalas (2011) who coupled three different calcium sources to three different signaling molecules: NMDAR to CaMKII, voltagesensitive calcium channels to calcineurin, and phosphodiesterase to calcium release. Adenylyl cyclase was coupled to both voltagesensitive calcium channels and NMDAR. The change in synaptic weight was calculated from kinase (tLTP) and phosphatase (tLTD) activity. This model investigated the role of cAMP degradation in triplet-based STDP, and showed that, if cAMP activity is spatially restricted to the membrane, the STDP profile is similar to that observed in cortical layer 2/3 slices. The STDP profile for spatially diffuse cAMP activity was consistent with that observed in hippocampal cell culture.

In the striatum, endocannabinoid production and activation of CB1Rs are required for most forms of tLTD (Mathur and Lovinger, 2012); thus, Cui et al. (2016) extended the signaling pathways from Graupner and Brunel (2007) with 2-arachidonoylglycerol; (2-AG the main endocannabinoid) production via mGluR- and M1R activation. Cui et al. (2016) utilized a single-compartment model of electrical activity of a spiny projection neuron for calcium dynamics, coupled with a model of signaling pathways underlying STDP in striatum, including calcium-induced calcium release from internal stores. This model used a combined 2-AG- and CaMKII-based plasticity rule, where the direction of plasticity (LTP or LTD) was determined by the product of the presynaptic weight (2-AGbased) and postsynaptic weight (CaMKII based). The strength of this model is the ability to show the mechanism whereby decreasing the number of pairings converts NMDAR-dependent tLTP to an endocannabinoid-dependent tLTP, which was confirmed experimentally (**Figure 5**; Cui et al., 2015, 2016). The underlying hypothesis of this model (that was confirmed experimentally) is that moderate activation of CB1R caused endocannabinoid-mediated tLTD whereas large CB1R activation leads to tLTP. In the model, 10–20 negative pairings trigger large endocannabinoid transients that result in endocannabinoidmediated tLTP. However, CB1R desensitization and partial depletion of calcium in the endoplasmic reticulum (CaER) starts to be significant after 20 pairings, so that CB1R activation is in fact smaller with more than 20 pairings than with 10–20 pairings. As a result, the expression of endocannabinoidmediated tLTP is restricted to 10–20 negative pairings, in agreement with experimental observations (Cui et al., 2015, 2016). On the other hand, as in the original model by Graupner and Brunel (2007), calcium levels become large enough to activate significant amounts of CaMKII only after 40–50 negative pairings, thus restricting the expression of NMDAR-dependent tLTP to this range of pairings. As a result, this model successfully reproduces the experimental observation that the endocannabinoid-mediated tLTP expressed at 10–20 positive pairings disappears, to be replaced by NMDAR-dependent tLTP after 50 pairings (**Figure 5**). The addition of presynaptic dopamine signaling to the model correctly predicted that the CB1R-dependent tLTP observed with 10–20 pairings is also under the control of presynaptic D2R (**Figure 5**).

# Exploring in Vivo-Like Conditions

One benefit of computational modeling is the ability to isolate a specific aspect of STDP and address the impact of this very aspect at the level of networks and/or learning in vivo. For instance, Kempter et al. (1998) used spike-based models to explore how the pulse structure of neuronal signals and events on a millisecond scale influenced learning rules. Clopath et al. (2010) utilized a voltage-based plasticity rule, consistent with a wide body of experimental data, to study the emergence from plasticity of connectivity patterns in a cortical network. Along the same line, variants of the classical computational STDP rule have been devised that yield broad synaptic weight distributions matching the available experimental observations (Gilson and Fukai, 2011).

However, one major detractor of STDP is its deterministic and constant spike timing (interval between spikes within a pairing) and inter-stimulation interval (interval between consecutive pairings), which diverge highly from biological variability. One of the most pressing questions of learning and memory is which stimuli resemble in vivo-like conditions best. Gjorgjieva et al. (2011) showed that a triplet model of STDP, depending on the interactions of three precisely timed spikes, described plasticity experiments closer to natural stimuli measured in the brain. Graupner et al. (2016) compared in silico plasticity outcomes to several types of irregular, in vivo-like, firing patterns to investigate the influence of firing rate and spike timing on synaptic plasticity. They showed that sensitivity of plasticity to spike-timing is reduced by adding jitter (irregularity) to spikepairs. Using physiological firing patterns recorded in awake behaving macaque monkeys, Graupner et al. (2016) further showed that moderate variation of firing rate, without any timing constraints, could reproduce synaptic changes induced by spike timing. This result offers a different view on the central role played by spike timing in long-term synaptic plasticity.

Most computational models of STDP indicate that plasticity disappears when the timing between pre and postsynaptic pairings loses its regularity. However, it is not clear what amount of noise can be tolerated for STDP or ITDP to be expressed (robustness) and whether this amount depends on the signaling pathway supporting the plasticity. This question has recently been tackled both experimentally and in a computational model, using noisy STDP stimulations where the timing between the pre- and the postsynaptic stimulations was jittered (Cui et al., 2018). As stated above, in striatum three forms of STDP are observed: NMDAR-tLTP, endocannabinoid-tLTD and endocannabinoid-tLTP (Cui et al., 2015, 2016). These three forms do not show similar sensitivity to jittered spike timing: NMDAR-tLTP appeared poorly resistant whereas endocannabinoid-plasticity (tLTD and tLTP) appeared more robust (Cui et al., 2018). Moreover, increasing the average pairing frequency or the number of pairings reinforces NMDAR-tLTP and increases resistance to jittered spike timing. These results suggest that the probability to observe the various forms of STDP in vivo is a multivariate function of the mean spike timing, the number of pairings, the frequency of pairings and also the variability of the spike timing. The shape of this multivariate function is thus more complex than e.g., a monotonic decay with increasing variability of the spike timing, and could reveal a functional specialization of each of these STDP forms to sub-regions of the stimulation train parameters.

# CONCLUSIONS AND FUTURE DIRECTIONS

In addition to the pre- and postsynaptic firing patterns, a third factor for STDP control comprises not only the classical neuromodulators (dopamine, noradrenaline or acetylcholine to name a few) but also neuropeptides (BDNF), unconventional neurotransmitters (NO) and astrocytes surrounding neurons, which can uptake or release neurotransmitters and neuromodulators. The spectrum of the third factor of STDP is even larger since it can be extended to neurotransmitters acting as neuromodulators such as GABA and glutamate (via their tonic component) and endocannabinoids. Here, we reviewed the main effects of the third factor on STDP: from the emergence of STDP, to the shaping of STDP i.e., the dependence on ∆tSTDP, and the magnitude and polarity of plasticity.

Beyond the time scale of ∆tSTDP that is consistently in the ∼80 ms range, the studies that explored STDP properties have used a large variety of pairing protocols to induce STDP. This diversity in stimulation protocol renders the comparison between studies exceedingly difficult. As described above, beside its dependence on ∆tSTDP, STDP expression is highly affected by varying the structure of STDP pairings (1:1, 1:2, . . . n:n or theta bursts; Edelmann et al., 2015), or the number and/or frequency of pairings (Sjöström et al., 2001; Cui et al., 2016) (for review see Sjöström et al., 2008; Feldman, 2012; Edelmann et al., 2017). It is thus expected that the effect of neuromodulation also would strongly depend on the STDP activity pattern (as an example see Edelmann et al., 2015).

How the local interneuron networks (GABAergic or cholinergic) or the neuromodulatory afferents are recruited and impact STDP, depends on the activity patterns of the two main inputs. i.e., the third factor effect may vary with or depend on a triplet of characteristics: ∆tSTDP, number of pairings, frequency of pairings. Optogenetics will most certainly be a key method to induce neuromodulator release in a more time-controlled manner to mimic for example phasic activity or explore precisely the retroactive action of neuromodulation on STDP properties.

The number of experimental studies investigating the signaling pathways underlying the STDP expression and their modulation by a third factor is still limited and needs further consideration. The signaling pathways underlying frequencydependent plasticity (triggered by high- or low-frequency stimulations) have been more thoroughly explored, but need to be fully address in STDP. Signaling though G-protein coupled receptors is far more complex than the static view of the list of proteins that compose each signaling pathways. For instance, G-protein coupled receptors exhibit the ''biased agonism,'' i.e., the notion that a given agonist of a signaling pathway activates only a subset of all the signaling pathways associated with its receptor (Kenakin and Christopoulos, 2013). In other words, two agonists of the same signaling pathway, even of the same receptor, activate different subsets of reactions, thus yielding different biological effects. One potential mechanism explaining biased agonism is the interplay between differential ligand-binding kinetics and the kinetics associated with different cell signaling processes (Klein Herenbrink et al., 2016). In this context, the subsets of signaling processes effectively activated by STDP pairings could differ from those activated by the stronger protocols employed in frequency-dependent plasticity.

The complexity of G-protein coupled receptor signaling also has consequences on STDP modulation. The available experimental data surveyed above point to a general rule according to which neuromodulation by monoamines or acetylcholine is mostly controlled by the type of G-protein coupled receptors activated: regardless of the agonist, Gi-coupled and Gq/11-coupled receptors favor tLTD, whereas Gs- and Golf-coupled receptor activation leads to tLTP. One might therefore erroneously conclude that two modulators would have the same effect by activating the same signaling pathway. This would of course be at odds with the concept that different neuromodulators exhibit different biological effects, due to different receptor affinities, different receptor locations, co-localization of diverse downstream signaling molecules, and the ability of phosphorylated receptors to switch their coupling to different G proteins. Hence, dopamine signaling via D1R may display different biological effects from noradrenaline signaling via β–adrenergic receptors, although both activate the Gs/Golf signaling pathway. Future computational models of STDP modulation should aim to reconcile the general scheme of the above rule with the specificity of neuromodulators, probably through variants of biased agonism.

Because of the complexity of the mode of action of neuromodulators, most of the studies have investigated the role of only a small number of neuromodulators one by one (the neuromodulator systems have mostly been activated or inhibited one-at-a-time), but the crosstalk between neuromodulators is critical, as demonstrated in only a few studies: for dopamine and acetylcholine (Brzosko et al., 2017), dopamine and GABA (Xu and Yao, 2010), dopamine and noradrenaline (Seol et al., 2007) or dopamine and endocannabinoids (Cui et al., 2015). The effects of other neurotransmitters/neuromodulators (such as adenosine, serotonin or endocannabinoids), or neuropeptides (substance P, enkephalins, oxytocin), fatty acids (arachidonic acid, cholesterol, omega-3), hormones or the role of other non-neuronal cells (astrocytes, oligodendrocytes, microglia, pericytes, ependymal cells or endothelial cells) remain to be investigated in STDP expression; Indeed, most of these actors are known to modulate rate-dependent plasticity. Furthermore, the effects of neuromodulators in STDP maintenance remain to be determined and not only for the induction phase of STDP. It has been shown in a rate-coded plasticity at CA1 hippocampal synapses that D1-like-receptor inhibition blocks late-phase LTP (Huang and Kandel, 1995), impedes consolidation of memory and accelerates its erasure (Wang et al., 2010; Lisman et al., 2011). Similarly, the third factor effect should be evaluated in the late phase of STDP (maintenance and potentially erasure).

By fully taking into account the third factor, i.e., a multicomponent learning rule, the computational power of neural networks might be considerably improved (as reviewed in Ku´smierz et al., 2017). Up to now, the third factor has usually been considered in isolation from the pre- and postsynaptic firing patterns. This experimental convenience might well disguise more complex network-level properties. In this regard, the fact that the level of tonic GABA in the local network can switch STDP from Hebbian to anti-Hebbian may have important consequences in dendritic computation and in a network context (Hiratani and Fukai, 2017). The interplay between changes of the firing rate of some of the network neurons due to Hebbian STDP and resulting changes in tonic GABA could give rise to abrupt STDP shifts locally from Hebbian to anti-Hebbian. Such local STDP shifts may provide the network with self-organizing properties that would not be predicted easily when the third factor is considered in isolation. Added to the fact that different synapse types in the network can have different STDP rules (and possibly, different modulation by the third factor), the complexity and variety of the resulting network dynamics would considerably increase. Note that here again, computational models will be instrumental to explore the potential impact of these mechanisms on the dynamics and functional properties of neural networks.

A fair criticism of the physiological relevance of STDP has been raised by Lisman et al. (2011) since in vivo the back-propagating action potential is obviously not triggered with a somatic current injection in the postsynaptic neuron (as classically performed in STDP experiments) but rather with the dynamic integration of synaptic inputs whose build-up would eventually reach the action potential threshold. Input-timingdependent plasticity (ITDP), a form of heterosynaptic plasticity, consists in paired activation of presynaptic inputs separated by an interval ∆tITDP, leading to sub- or suprathreshold activity in the postsynaptic neuron (Dudman et al., 2007; Williams et al., 2007). Therefore, ITDP could be viewed as an attractive naturalistic upgrade of STDP, not only for experimental studies (Dudman et al., 2007; Cho et al., 2011; Mehaffey and Doupe, 2015; Brandalise et al., 2016; Leroy et al., 2017) but also for computational models (Shim et al., 2016). ITDP has been reported in amygdala following activation of thalamic and cortical inputs (Humeau et al., 2003; Cho et al., 2011), in hippocampal CA1 (Dudman et al., 2007), CA2 (Leroy et al., 2017) or CA3 (Brandalise et al., 2016) pyramidal cells and in avian basal ganglia (Mehaffey and Doupe, 2015). Interestingly, GABA and enkephalin have been shown to modulate CA2 hippocampal ITDP (Leroy et al., 2017), which paves the way for future studies investigating the role of the third factor in ITDP properties.

The vast majority of STDP studies investigating the third factor have been achieved ex vivo (cell cultures or acute brain slices), although few studies have addressed neuromodulation of STDP in vivo (Mu and Poo, 2006; Schulz et al., 2010; Cassenaer and Laurent, 2012; Yagishita et al., 2014; Fisher et al., 2017). In ex vivo studies, neuromodulators (dopamine, acetylcholine) are typically applied exogenously because of their very low levels when compared to in vivo. Neuromodulators are released in tonic and phasic modes in vivo and therefore ex vivo bath-applications of neuromodulators or specific agonists hardly mimic such complexity of the neuromodulation. It would be important to explore the in vivo neuromodulation needed to stabilize STDP or ITDP, by transforming eligibility traces into plasticity, and thus allowing an activity pattern sequence to be pertinent for the engram. Thus, there is a need to collect data in vivo in awake and behaving animals and model in vivo-like plasticity rules and stimulation patterns to fully understand the action of the third factor in Hebbian learning and information storage and recall.

#### REFERENCES


### AUTHOR CONTRIBUTIONS

AF and AM participated in the writing of the ''Neuromodulators Affecting the Expression Polarity and Shape of STDP'' section. JJ-S, HB and KB wrote the ''Molecular Pathway-Based Computational Models of STDP'' section. SV wrote the ''Modulation of STDP by Astrocytes: the Forgotten Third Factor'' section. LV wrote the ''Introduction, Neuromodulators Affecting the Expression Polarity and Shape of STDP and Conclusion'' sections. All authors have edited and corrected the manuscript.

#### FUNDING

This work was supported by grants from the Agence Nationale pour la Recherche (ANR-CRCNS Dopaciumcity) and the LabEx Paris-Sciences et Lettres (PSL). AM was supported by the Fondation pour la Recherche Médicale (FRM). KB and JJ-S were supported by the joint NIH-NSF CRCNS program through NIAAA grant R01DA03889.


GABAergic transmission and synaptic plasticity in the prefrontal cortex. Proc. Natl. Acad. Sci. U S A 106, 5942–5947. doi: 10.1073/pnas.0811431106


tonic GABAergic signaling in striatum. Neuropharmacology 121, 261–277. doi: 10.1016/j.neuropharm.2017.04.012


structural plasticity of dendritic spines. Science 345, 1616–1620. doi: 10.1126/ science.1255514


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Foncelle, Mendes, Je˛drzejewska-Szmek, Valtcheva, Berry, Blackwell and Venance. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Emergence of Relaxation Oscillations in Neurons Interacting With Non-stationary Ambient GABA

Denis A. Adamchik\*, Valery V. Matrosov and Victor B. Kazantsev

Lobachevsky State University, Nizhny Novgorod, Russia

Dynamics of a homogeneous neural population interacting with active extracellular medium were considered. The corresponding mathematical model was tuned specifically to describe the behavior of interneurons with tonic GABA conductance under the action of non-stationary ambient GABA. The feedback provided by the GABA mediated transmembrane current enriched the repertoire of population activity by enabling the oscillatory behavior. This behavior appeared in the form of relaxation oscillations which can be considered as a specific type of brainwaves.

Keywords: neural oscillation, rate model, tonic conductance, GABA, interneurons

## 1. INTRODUCTION

#### Edited by:

Jean-Marie Charles Bouteiller, University of Southern California, United States

#### Reviewed by:

Clayton Bingham, University of Southern California, United States Werner Kilb, Johannes Gutenberg-Universität Mainz, Germany

#### \*Correspondence:

Denis A. Adamchik denis.adamchik@gmail.com

Received: 25 January 2018 Accepted: 12 March 2018 Published: 05 April 2018

#### Citation:

Adamchik DA, Matrosov VV and Kazantsev VB (2018) Emergence of Relaxation Oscillations in Neurons Interacting With Non-stationary Ambient GABA. Front. Comput. Neurosci. 12:19. doi: 10.3389/fncom.2018.00019 Historically, the focus of experimental and theoretical studies of brain signaling was almost exceptionally on neurons and their networks. Being the only electrically excitable cells in the nervous system, neurons are able to communicate by receiving, processing and generating electrical signals in the form of spike trains (Nicholls et al., 2001). All other structures constituting the nervous tissue such as glial cells and extracellular matrix (ECM) until very recent decades were not taken into account in the mechanisms of information processing.

Glial cells and various exctracellular structures were primarily thought to perform a number of auxiliary functions such as trophic, supportive and immune (Allen et al., 2009). The comprehension of inalienability of glia and ECM to the neuronal signaling came with the discovery of chemical synaptic transmission machinery (Krnjevic, 1974 ´ ) and secretory function of astrocytes (Martin, 1992). Glia turned out to be a gigantic chemical factory of the nervous system, governing neurons and using the extracellular space as an intermediary (Barres, 1991).

At present, there has been a great number of theoretical and experimental studies devoted to neuron-glia interaction (Bezzi and Volterra, 2001). One of the most prominent concepts in the field was that of the tripartite synapse (Araque et al., 1999). Glial cells, particularly astrocytes, can effectively influence and modulate the synaptic transmission. Many aspects of such modulations were discussed in a number of computational studies (Postnov et al., 2007; Gordleeva et al., 2012; Kazantsev et al., 2012; Volman et al., 2012; Lazarevich et al., 2017).

Besides several glial cell types, the extracellular space itself can be an important player in neuronal signaling. It serves as an interstitial transport system mediating cell-to-cell communications by means of numerous active chemicals (Sykovaá and Nicholson, 2008). This type of communications is called "volume" transmission and is characterized by signal diffusion in a three-dimensional fashion within the brain extracellular fluid (Agnati et al., 1995). The "volume" transmission depends crucially upon the actual geometry of the ECS (Syková, 2004) which has great relevance for pharmacokinetics and actions of neuropsychoactive drugs (Zoli et al., 1999).

One of the major neurotransmitters in the CNS is γ - Aminobutyric acid (GABA) (Webster, 2001). It mediates intercellular communications by participating in both "wiring" and "volume" transmission (Semyanov et al., 2004). The "wiring" action of GABA is through mediating the synaptic transmission by activating the postsynaptic (phasic) GABAA-receptors. The "volume" transmission is, in its turn, carried out by "overspilled" ambient GABA which regulates neuronal excitability by creating the extra transmembrane current through extrasynaptic (tonic) GABAA-receptors.

GABA was reported to maintain the fast neuronal oscillations (gamma, 20–80 Hz) in inhibitory interneuron networks (Whittington et al., 1995; Bartos et al., 2007). In the computational study (Wang and Buzsáki, 1996), GABA<sup>A</sup> synaptic transmission was shown to provide a suitable mechanism for synchronized gamma oscillations in a sparsely connected network of fast-spiking interneurons. Incidentally, GABA was reported to enhance collective behavior in neuronal axons (Traub et al., 2003). Specifically, gammafrequency oscillations were demonstrated to coexist with phasic high-frequency oscillations (>90 Hz) in principal cell axon populations.

Primarily, GABA was considered to be the main inhibitory neurotransmitter in the brain until it was shown experimentally to be able to perform the bi-directional regulation of neuronal spiking activity (Song et al., 2011). Based on this experimental finding, a number of mathematical models describing the action of ambient GABA on the excitability properties of interneurons were suggested (Adamchik et al., 2015; Adamchik and Kazantsev, 2017). In Adamchik et al. (2015), the behavior of a single interneuron embedded in the extracellular space with constant ambient GABA concentration was studied. It was shown that depending on the parameters of tonic current, such as tonic conductance density and GABA reversal potential, the interneuron demonstrated different behavioral modes including self-oscillations. The impact of stationary GABA at the population level was studied subsequently in Adamchik and Kazantsev (2017). Specifically, it led to bistability between asynchronous firing and zero-activity state.

In this paper, we study the effects of non-stationary, activity dependent GABA upon population dynamics of interneurons. To this end, we propose a mathematical model accounting for the feedback between interneurons and ambient GABA (section 2). The origin of the feedback has the following explanation. Extracellular GABA creates the additional transmembrane current through activation of extrasynaptic (tonic) GABAAreceptors. This current further changes the firing properties of interneurons (Adamchik et al., 2015), which immediately affects the synaptic release of GABA (Destexhe et al., 1994). Since extracellular GABA concentration depends, among others, on spillover, i.e., the diffusion of the neurotransmitter out of the synaptic cleft (Semyanov et al., 2004), it changes, which futher affects tonic conductance and provides the respective feedback.

Based on these considerations, we proposed a mathematical model using the following assumptions. First, we considered a particular case of a homogeneous population of interneurons which allowed us to describe their collective behavior using the simple rate-based formalism. Second, we neglected any spatial gradient of neurotransmitter, considering its concentration to be uniformly distributed over the entire extracellular space. This assumption allowed us to build the minimal model of the feedback avoiding dealing with an explicit model of spatiotemporal GABA dynamics. The model consisted of two coupled equations, one of which described the timecourse of population activity (section 2.1) while the other– the concentration of ambient GABA (section 2.2). The model was explored both numerically and analytically (section 3). The results including the appearance of relaxation oscillations were discussed in (section 4).

#### 2. MATERIALS AND METHODS

#### 2.1. Population Dynamics

Within the framework of rate-based formalism, a homogenous population of neurons is described by a single variable, e.g., the population activity, A. The rate of change of A is determined by the so called gain fuction gλ(I), which is unique for each cell type. The respective equation reads:

$$
\pi\_m \frac{dA}{dt} = -A + \mathfrak{g}\_{\lambda}(I) \tag{1}
$$

where τ<sup>m</sup> is the membrane time constant and I is the total input current an arbitrary neuron receives from the entire network. The latter is linearly dependent on population activity, I = JA, where the proportionality factor, J, is called coupling strength (Gerstner et al., 2014).

The gain gλ(I) is primarily a function of input current, I, but can also depend on a number of factors, collectively denoted here by λ. In our case, these are tonic conductance density, G, and GABA reversal potential, E, i.e., λ = (E,G).

The exact form of the gain function can be derived analytically only for a few simple neuron models, such as, for example, the quadratic integrate-and-fire (QIF), whose dimensionless normal form reads: v˙ = v <sup>2</sup> <sup>+</sup> <sup>κ</sup>. Using separation of variables and integration over infinite potential bounds, one can get:

$$\lg\_{\lambda}(I) = \frac{1}{\mathfrak{r}\_r + \mathfrak{r}\_m \kappa^{-1/2}} \tag{2}$$

where the dimensionless parameter κ depends both on input current I and other factors. The absolute refractory period, τr , is added to the period of oscillations to prevent the firing frequency from taking an arbitrarily large value. Note, that formula (Equation 2) is valid only for positive κ; when κ < 0 no oscillations occur and, as a result, gλ(I) = 0.

The gain function (Equation 2) describes qualitatively the responce of Class I excitability neurons. In these neurons, the transition from resting to spiking occurs via saddle-node on invariant circle bifurcation (SNIC), that allows them to fire with arbitrarily small frequency (Izhikevich, 2007). The Wang-Buzsa´ki interneuron (Wang and Buzsáki, 1996) belongs exactly to this type of neurons. In Adamchik and Kazantsev (2017), the original conductance-based model, modified in a way to account

TABLE 1 | Model constants.


for the additional transmembrane tonic current, was reduced to the QIF neuron. The dimensionless parameter κ took the following form:

$$\kappa = -\frac{1 + \langle G/G\_m \rangle^2}{4} + \frac{k}{G\_m^2} [I + G(E - E\_m)] \tag{3}$$

where Em, G<sup>m</sup> and k are constants listed in **Table 1**. The details of the reduction can be found in Adamchik and Kazantsev (2017).

Equation (1) along with the relations (Equations 2, 3) describes the time-course of population activity. It contains parameters such as coupling strength, J, and GABA reversal potential, E, which can take arbitrary values but remain unchanged. Tonic conductance density, G, is on the contrary a variable, which depends on local GABA concentration, C. The form of this dependence can be determined using a common kinetic formalism (Destexhe et al., 1998). According to a simplified kinetic scheme of the GABAA-receptor, which is assumed to exist in two conformations, open (O) and closed (C), one can get:

$$G = \bar{G}\frac{\alpha \mathcal{C}}{\alpha \mathcal{C} + \beta} \tag{4}$$

where α and β are forward (activation) and backward (deactivation) rates, respectively, and G¯ is maximum conductance density. Rate values were taken from Koch and Segev (1998) and are given in **Table 1** for reference.

Note, that equation (4) does not determine a momentary but rather a stable-state value of G. Nevertheless, we may use it because conductance relaxation time τ<sup>G</sup> = 1/(αC + β) ≤ 1/β ≈ 5 ms, which is far less than the operating time of ambient GABA concentration which amounts to hundreads of milliseconds (Semyanov et al., 2004).

#### 2.2. Ambient GABA Dynamics

According to Semyanov et al. (2004), extracellular GABA concentration is regulated by uptake, non-synaptic release and spillover. Uptake is carried out by GABA transporters which decrease the concentration by binding and removing GABA molecules from the extracellular space. Ambient GABA can originate from various sources. It can escape from synaptic cleft (spillover) and can be released via non-vesicular mechanism by neurons and glia. Both spillover and non-synaptic release increase ambient GABA concentration but do it differently. Unlike non-synaptic release, spillover depends crucially on synaptic dynamics and, as a result, on population activity. These general considerations allowed us to write a governing equation for ambient GABA concentration:

$$\frac{dC}{dt} = -\frac{C - C\_0}{\tau\_C} + S(A) \tag{5}$$

The first term is supposed to describe the mutual action of uptake and non-vesicular release, which counterbalance each other by maintaining an optimal background GABA level, denoted here by C0. Spillover is described, in its turn, by the second term, S(A), representing the production function of GABA and depending explicitly on population activity. The exact form of the production function was chosen in a way to describe qualitatively correctly the properties of synaptic neurotransmitter release. In the most common case, the production function reads as follows:

$$S(A) = Q \frac{A \text{tr}\_P}{A \text{tr}\_P + 1} \tag{6}$$

where τ<sup>P</sup> is GABA production time constant and Q is the maximum production rate.

The exact form of the production function can be derived based on the following consideration. Let δC<sup>m</sup> be the amount of GABA released in response to a single spike. Then, due to exhaustion of synaptic vesicle pools, the next spike, coming τ time units after the first one, will evoke the release of a lesser amount of neurotransmitter, precisely δC = δCm[1 − exp(−τ/τP)]. For a Poisson spike train, the interspike interval distribution (ISI) with the mean firing rate equal to A reads: P(τ ) = A· exp(Aτ ). Then, the average amount of GABA released in response to a spike from the spike train will be hδCi = R <sup>∞</sup> 0 δC· P(τ )dτ = δCm/(1+AτP). The product A·hδCi gives the required production rate (cf. 6), where Q = δCm/τP.

#### 2.3. Dynamical System

Two coupled ordinary differential equations (ODE) (1, 5) with relations (2–4, 6) form a 2D dynamical system. Its state variables are population activity, A, and ambient GABA concentration, C. Besides some constants (see **Table 1**), the equations contain a number of free parameters, which can roughly be split into two distinct groups. The first group, (E,G¯ , J), consists of the parameters controlling population activity, while the second one, (C0, Q), determines ambient GABA concentration. Our task is to reveal how the dynamics of equations (1, 5) depend on all these parameters. Some preliminary considerations concerning the matter are the following.

In absense of the second equation (Equation 5), the system reduces to a simple 1D phase line, corresponding to the case of stationary external medium. This particular case was a subject of our previous study (Adamchik and Kazantsev, 2017). It was shown then that introduction of tonic current did not lead to any new dynamical effects compared to the reference case, (G = 0), characterized by a simple stable-state dynamics. It resulted, however, in appearance of a monostable regime of asynchronous firing once tonic current parameters, E and G, were properly tuned, specifically, in a way that the point (E,G) was located above a certain curve on the parameter plane. This curve was shown previously to be a border of the self-oscillatory mode in the model of a tonically driven single neuron (Adamchik et al., 2015). The impact of coupling strength, J, on population activity consisted, in its turn, in inducing bistability, i.e., coexistence of resting and asynchronous firing, and expanding the bistability region at the cost of trivial (zero-activity) monostable solutions.

In this study our particular interest is on the feedback-induced dynamical effects. For this purpose we specifically focus on the parameters governing ambient GABA concentration, i.e., C<sup>0</sup> and Q. Note that such parameters as baseline concentation, C0, and maximum production rate, Q, can be controlled pharmacologically in experiments.

#### 2.4. Semi-explicit Model

To verify the validity of our population model prediction as well as to visualize neural oscillations we built a semi-explicit computational model of the respective spiking network. To do this we replaced the equation for the population activity (Equation 1) with an explicit spiking neural network model but preserved (Equation 5), which describes the dynamics of ambient GABA.

We considered specifically a network of N = 100 interneurons randomly coupled with probability p = 0.1. Each neuron was described by the original conductance-based model (Wang and Buzsáki, 1996) with the additional tonic current term: IGABA = G(u − E), where u is the membrane potential. Tonic conductance density, G, depended here on extracellular GABA concentration just in accordance with equation (4). Synaptic transmission was mediated by phasic GABAA-receptors. The total synaptic current to an arbitrary neuron was determined as the normalized sum over the contributions of all its presynaptic neighbors:

$$I\_{\rm syn} = \frac{1}{M} \sum G\_{\rm sym} r(\mu - E) \tag{7}$$

where M is the mean number of presynaptic inputs: M = Np. The fraction of the receptors in the open state, r, obeyed the kinetic equation

$$\frac{dr}{dt} = \alpha \, T(1 - r) - \beta r \tag{8}$$

while the synaptic concentration of GABA, T, strictly followed the potential at the presynapse:

$$T(\mu\_{pre}) = \frac{T\_{\text{max}}}{1 + \exp\left(-\frac{\mu\_{pre} - \Theta}{\Delta}\right)}\tag{9}$$

Besides transmembrane and synaptic components, the total current to each neuron included also a constant one, I0, which served to regulate the level of depolarization. The parameters of synaptic transmission are listed in **Table 2**.

To couple the explicit spiking network model with the equation describing ambient GABA dynamics we calculated at



each time step the instantaneous population activity, using for averaging the α-function (Dayan and Abbott, 2001):

$$A(t) = \frac{1}{N} \sum\_{\substack{s=j\mathbf{r}\_s, \\ j \in N}}^{3\mathbf{r}\_w} \alpha(\mathbf{s}) \, \mathbf{S}(t-s; t-s+\mathbf{r}\_s), \tag{10}$$

$$\alpha(s) = \left[ \frac{s}{\tau\_w^2} \exp\left(-\frac{s}{\tau\_w}\right) \right]\_+ \tag{11}$$

Here, S(t − s;t − s + τs) is the total number of spikes, the entire network generates within the respective time window; τ<sup>w</sup> = 20 ms and τ<sup>s</sup> = 1 ms are averaging and sliding windows, respectively.

### 3. RESULTS

#### 3.1. Numerical and Phase Plane Analysis

First, we performed numerical analysis of the dynamical system (Equations 1, 5). We carried out numerous simulation trials for a wide range of biologically relevant parameter values. In each trial, the system proceeded with the same initial conditions (A|t=<sup>0</sup> = 0, C|t=<sup>0</sup> = C0) corresponding to zero population activity and baseline GABA concentration, respectively. We found out that, depending on parameters, the system demonstrated either oscillatory or stationary behavior (see **Figure 1**). The parameters of the simulations are listed in **Table 3**.

Different kinds of behavior can be accounted for using the phase plane. In case of oscillations, the trajectory first makes a big loop before converging to the limit cycle (see **Figure 1B**). The latter is intersected by the A-nullcline, coinciding for the small A's with the border between zero and non-zero gain: gλ(0) = 0 (see **Figure 2B**). It means that the system in the oscillatory mode sequentially visits the region of excitatory GABA action. The oscillations would evidently not occur if the baseline GABA concentration, C0, exceeded the borderline value between inhibition and excitation, C+. In other words, if the system was placed into the region of inhibitory GABA, it would never leave it. In the stationary mode, the trajectory moves up slower than the A-nullcline does (see **Figure 1D**), so it converges to the fixed point corresponding to stationary asynchronous firing. This scenario realizes if the maximun production rate, Q, is lower than a certain threshold value. These considerations helped us subsequently to determine conditions for oscillations (section 3.2).

FIGURE 1 | The time-course of population activity (solid) and ambient GABA concentration (dashed) along with the respective phase plane trajectories (thick solid) for oscillatory (A,B) and stationary (C,D) modes. On the phase portraits (B,D), thin gray lines correspond to A- (solid) and C- (dashed) nullclines, respectively. Separatrices of the saddle are represented by a pair of thin black lines with arrows indicating the flow direction.

TABLE 3 | Model parameters.


Let us now describe the biophysical mechanism underlying periodical oscillatory solutions (see **Figure 1B**). If the baseline ambient GABA concentration, C0, is high enough to make neurons fire but not too high to inhibit them by shunting, the initially silent neurons start firing. Non-zero population activity makes activity dependent ambient GABA concentration steadily grow up through synaptic release and spillover (OM). If parameters, governing ambient GABA dynamics, are properly tuned, then, at some point, tonic GABA switches from excitation to inhibition. On the phase plane, it corresponds to the intersection of the trajectory with the A-nullcline (M). As soon as the intersection occurs, the gain becomes zero and population activity starts decreasing to its steady-state (zero) value with a time constant of the membrane, τm. While population activity goes down, the concentration keeps growing but its grow rate gradually slows down (MN). Eventually, the rate of change of C becomes negative and the trajectory moves down with a time constant of concentration, τ<sup>C</sup> (NO′ ). At some point (O′ ), the trajectory re-enters the region of excitatory GABA and the entire process starts from the scratch. Note, that concentration does not reach its baseline level, C0, so the magnitude of the limit cycle is less than that of the initial loop.

To avoid the trajectory from making a loop before converging to the limit cycle, we took the initial conditions exactly at

the upper border between inhibitory and excitatory GABA: A|t=<sup>0</sup> = 0, C|t=<sup>0</sup> = C+. The condition for the border follows directly from equation (1) as: gC+−0(0) > 0, gC++0(0) = 0. Based on the explicit analytical expression for the gain function (Equations 2, 3) as well as on the relation between tonic conduction density, G, and GABA concentration, C, (Equation 4), one can get:

$$\mathcal{C}\_{\pm} = \frac{\beta}{\alpha} \frac{G\_{\pm}}{\bar{G} - G\_{\pm}},\tag{12}$$

where

$$G\_{\pm} = G\_m \left( \times \pm \sqrt{\mathbf{x}^2 - 1} \right), \ \mathbf{x} = \frac{2k}{G\_m} (E - E\_m) \tag{13}$$

The minus-subscripted concentration, C−, corresponds to the transition from inhibition to excitation as we move upwards the C-axis and is given here just for reference. Its value for the actual choice of parameters (see **Table 3**) is negligible and its existence does not play any substantial role for oscillations. Oscillations occur essentially at the border between excitatory and inhibitory GABA and not vice versa.

Note, that the solution exists only if GABA reversal potential, E, lies above a certain threshold, whose value is determined by zero determinant condition (see equation (13)): E <sup>∗</sup> = E<sup>m</sup> + Gm <sup>2</sup><sup>k</sup> ≈ −56.8 mV, which exceeds the resting membrane potential (≈ −64 mV) by 9.2 mV.

The time-course of the dynamical variables as well as the shape of the limit cycle (see **Figure 2**) are typical for relaxation oscillations. In the excitatory region, the population activity relaxes to the value determined by the gain function but as soon as it leaves it, A starts the exponential decay to zero. The concentration follows the population activity with a delay caused by the difference between the time constants of membrane, τm, and of concentration, τC.

# 3.2. Conditions for Oscillations and Their Characterisctics

Our next step was to determine the region in the parameter plane (C0, Q) where the system had periodical solutions. For this purpose we implemented the following calculations. For each point (C0, Q) from the rectangular (0, Cmax) × (0, Qmax) we traced the time evolution of the dynamical system Equations (1, 6) under the same initial conditions: A|t=<sup>0</sup> = 0, C|t=<sup>0</sup> = C+. Depending on the parameters, the system either remained at the starting point or relaxed to the upper stable state, or oscillated periodically. The periodicity was ascertained based on multiple crossings of the trajectory with the A-nullcline. The border between the regions of oscillatory and transient solutions is depicted in **Figure 3** for two different values of GABA reversal potential. Specifically, **Figure 3A** corresponds to highly depolarizing GABA, while **Figure 3B** corresponds to GABA, whose reversal potential is slighly above (precisely by 1.8 mV) the oscillatory threshold.

The oscillatory region for highly depolarizing GABA has roughly the shape of a semi-infinite strip, (0, C+) × (Q−,∞), where the border values C<sup>+</sup> and Q<sup>−</sup> are both dependent on the reversal potential, E. As far as we get closer to the oscillatory threshold, the region shrinks until collapsing at E = −56.8 mV. The shape of the oscillatory region for high E's implies that oscillations occur if both the baseline GABA concentration and the GABA production rate are located below and above their respective threshold values: C < C+, Q > Q−. Note, that the upper bound C<sup>+</sup> corresponds exactly to the concentration value at the starting point of simulation. The shape of the oscillatory region has the straightforward phase-plane interpretation (see section 3.1). Although, we managed to find the explicit analytical expression for C<sup>+</sup> (see Equations 12, 13), there was no way to obtain such for Q−(E), the more so it depends not only on E but on C<sup>+</sup> as well, which is illustrated in **Figure 3B**.

Having found the oscillatory region we looked for the magnitude and period of oscillations as functions of C<sup>0</sup> and Q. To this end we started time-course simulations from the same point at the limit cycle as we did before in oder to pass the transition phase. We defined the period of oscillations as the time before two subsequent intersections with the A-nullcline with the same sign of the slope. The magnitude of oscillations was determined as the maximum value of population activity and concentration, respectively. The results are depicted in **Figure 4**. First, we fixed the maximum production rate, Q, and found

numerically the dependence of both period and magnitude on the baseline concentration of GABA, C<sup>0</sup> (see **Figures 4A,B**). Next, we fixed C<sup>0</sup> and found the respective dependencies on the parameter Q; the corresponding graphs are depicted in **Figures 4C,D**. Note, that the period grows infinitely as soon as we approach the boundaries of the oscillatory region. The dependence of the magnitude on the production rate is more drastic than that on the background GABA level. The explanation is quite simple – the higher Q is, the faster the periodic trajectory intersects the A-nullcline, the lower is the magnitude.

#### 3.3. Spiking Network Simulation

The results of our computer simulation are depicted in **Figure 5**. Note that they quite correctly reproduce those obtained using the original rate model. In case of periodicity, after an initial burst of population activity, the network demonstrates stationary oscillatory behavior (**Figure 5C**) just in accordance with the prediction (**Figure 1A**). The stationary-like behavior in the spiking network model was another option that we can verify at the network level (see **Figure 5D**).

# 4. DISCUSSION

We proposed a self-consistent model of interneurons interacting with extracellular, activity dependent GABA. The model represented two coupled nonlinear ODEs describing the dynamics of population activity and ambient GABA concentration, respectively. To write the first equation we used the well-known Wilson-Cowan formalism describing the low-pass behavior of a neural ensemble (Gerstner et al., 2014). The gain function was chosen in a way to properly mimic the behavior of the interneuron with tonic GABA conductance (Adamchik and Kazantsev, 2017). The dynamics of ambient neurotransmitter were quantitatively accounted for on the basis of empirical evidence about the sources and sinks of extracellular GABA (Semyanov et al., 2004). Mathematically, the model was a continuous-time dynamical system on a plane. It was shown to admit both stationary and periodic solutions depending on the parameters governing neurotransmitter concentration. Unlike the stationary-like behavior, periodicity was a feedbackinduced feature with clear biophysical explanation. In oscillatory mode, the system evolved between the regions of excitatory and inhibitory GABA. In each of these regions the dynamical variables relaxed to their stable-state values, so that the type of the oscillations was essentially relaxational. Such a pattern of synchronized population activity can be regarded as a specific type of brainwave.

We determined the conditions for oscillations and their characteristics such as period and magnitude as a function of GABA parameters. In particular, we found out that oscillations were possible only for strongly depolarizing GABA. For interneurons, GABA reversal potential had to exceed the RMP by at least 9.2 mV for oscillations to occur, which is above the reported values (Michelson and Wong, 1991; Verheugen et al., 1999; Chavas and Marty, 2003; Banke and McBain, 2006). The oscillatory region on the plane of baseline GABA concentration and maximum GABA production rate had roughly the shape of a semi-infinite stripe, i.e., there existed an upper background level of GABA and a lower intensity of its production, beyond which no oscillations occured. As a consequence, neural oscillations could be induced or suppressed pharmacologically, by changing GABA control parameters. This might have some biomedical implications since extrasynaptic GABA is believed to contribute to epileptic or schizophrenic brain activity (Brickley and Mody, 2012).

In conclusion, we need to mention the limitations of our present consideration. The suggested model of the feedback between neurons and extracellular GABA is minimal in the sense that it does not account for many key features of real neural networks and their environment. For example, when discussing ambient GABA dynamics (section 2.2) we assumed GABA sources, sinks, and receptors to be co-local. This allowed us to describe the time evolution of ambient GABA concentration with a simple ODE instead of building a detailed model accounting for the actual geometry of the extracellular space. Futher, we assumed uptake and non-vesicular release independent on population activity, although there is experimental evidence that this traditional view was too simplistic (Richerson and Wu, 2003).

## REFERENCES


In addition, we considered the special case of a homogeneous neural network which is a rough representation of real neuronal ensembles. Taking into view all these considerations, we must admit that our conclusions can offer only primary insights into the feedback-induced dynamics of GABA-driven interneurons. At the same time, they can be regarded as reference point for future studies applying more sophisticated methods.

#### AUTHOR CONTRIBUTIONS

DA designed the model and the computational framework, carried out the implementation and wrote the manuscript. VM contributed to the interpretation of the results. VK conceived the study and was in charge of overall direction and planning. All authors provided critical feedback and helped shape the research, analysis and manuscript.

### FUNDING

This research was supported by the Ministry of Education and Science of Russian Federation. Agreement (contract) No. 14.578.21.0107, unique project identifier: RFMEFI57815X0107.


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer CB and handling Editor declared their shared affiliation.

Copyright © 2018 Adamchik, Matrosov and Kazantsev. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Resonance Analysis as a Tool for Characterizing Functional Division of Layer 5 Pyramidal Neurons

Melvin A. Felton Jr. <sup>1</sup> \*, Alfred B. Yu<sup>2</sup> , David L. Boothe<sup>2</sup> , Kelvin S. Oie2† and Piotr J. Franaszczuk 2,3†

*<sup>1</sup> Computational and Information Sciences Directorate, U. S. Army Research Laboratory, Adelphi, MD, United States, <sup>2</sup> Human Research and Engineering Directorate, U. S. Army Research Laboratory, Adelphi, MD, United States, <sup>3</sup> Department of Neurology, Johns Hopkins University School of Medicine, Baltimore, MD, United States*

Evidence suggests that layer 5 pyramidal neurons can be divided into functional zones with unique afferent connectivity and membrane characteristics that allow for post-synaptic integration of feedforward and feedback inputs. To assess the existence of these zones and their interaction, we characterized the resonance properties of a biophysically-realistic compartmental model of a neocortical layer 5 pyramidal neuron. Consistent with recently published theoretical and empirical findings, our model was configured to have a "hot zone" in distal apical dendrite and apical tuft where both high- and low-threshold Ca2<sup>+</sup> ionic conductances had densities 1–2 orders of magnitude higher than anywhere else in the apical dendrite. We simulated injection of broad spectrum sinusoidal currents with linearly increasing frequency to calculate the input impedance of individual compartments, the transfer impedance between the soma and key compartments within the dendritic tree, and a dimensionless term we introduce called resonance quality. We show that input resonance analysis distinguished at least four distinct zones within the model based on properties of their frequency preferences: basal dendrite which displayed little resonance; soma/proximal apical dendrite which displayed resonance at 5–23 Hz, strongest at 5–10 Hz and hyperpolarized/resting membrane potentials; distal apical dendrite which displayed resonance at 8–19 Hz, strongest at 10 Hz and depolarized membrane potentials; and apical tuft which displayed a weak resonance largely between 8 and 10 Hz across a wide range of membrane potentials. Transfer resonance analysis revealed that changes in subthreshold electrical coupling were found to modulate the transfer resonant frequency of signals transmitted from distal apical dendrite and apical tuft to the soma, which would impact the frequencies that individual neurons are expected to respond to and reinforce. Furthermore, eliminating the hot zone was found to reduce amplification of resonance within the model, which contributes to reduced excitability when perisomatic and distal apical regions receive coincident stimulating current injections. These results indicate that the interactions between different functional zones should be considered in a more complete understanding of neuronal integration. Resonance analysis may therefore be a useful tool for assessing the integration of inputs across the entire neuronal membrane.

Keywords: resonance, layer 5 pyramidal neuron, functional division, electrical coupling, association, ionic conductance

#### Edited by:

*Vito Di Maio, Istituto di Scienze Applicate e Sistemi Intelligenti Eduardo Caianiello (CNR), Italy*

#### Reviewed by:

*Kalyana Srinivas Vadduri, Columbia University, United States Petr Lansky, Academy of Sciences of the Czech Republic (ASCR), Czechia*

\*Correspondence:

*Melvin A. Felton Jr. melvin.a.felton.civ@mail.mil*

*†Co-senior author.*

Received: *16 February 2018* Accepted: *16 April 2018* Published: *03 May 2018*

#### Citation:

*Felton MA Jr, Yu AB, Boothe DL and Oie KS and Franaszczuk PJ (2018) Resonance Analysis as a Tool for Characterizing Functional Division of Layer 5 Pyramidal Neurons. Front. Comput. Neurosci. 12:29. doi: 10.3389/fncom.2018.00029*

# INTRODUCTION

Neocortical architecture facilitates association-based information processing where feedforward and feedback signals connect the many different processing stages of the neocortex. The associative nature of neocortical function can even be observed on the scale of single neurons, such as layer 5 pyramidal neurons that play a central role in the functioning of neocortical microcircuits (Larkum et al., 1999; Larkum, 2013). The neuronal membrane of these neurons has a large spatial extent and is usually spread throughout all neocortical layers, and different parts of the neuron (e.g., proximal vs. distal) receive inputs from different regions of the brain (Larkum et al., 2009). Direct feedforward projections, which are typically of local origin, synapse close to the soma on proximal dendrites; on the other hand, feedback projections, which tend to originate from far away sources, like non-specific thalamocortical neurons or distant cortical neurons, synapse on the distal regions of the apical dendrite and apical tuft (Spruston, 2008; Hawkins and Ahmad, 2016). The perisomatic and distal apical regions of layer 5 pyramidal neurons have been identified as two distinct "zones" that both mediate action potential initiation (Larkum, 2013). Moreover, because these two spike initiation zones are electrically coupled, the pyramidal neuron is able to detect coincident feedforward input to its perisomatic regions and feedback input to its distal apical regions.

The concept of distinct functionally-defined zones has been expanded by taking into account additional functional aspects, such as the processes of synaptic integration within the large and complex membrane of layer 5 pyramidal neurons (Williams and Stuart, 2003; Polsky et al., 2004; Spruston and Kath, 2004). For instance, Spruston and Kath (2004), proposed a three-layer model of synaptic integration where: (1) basal/oblique dendrites and apical tuft are two distinct zones that collectively comprise an input layer, (2) proximal apical dendrite/soma and distal apical dendrite are two distinct zones that collectively comprise an integration layer, and (3) the axon hillock itself constitutes an output layer.

Previously, resonance analysis has been shown to be a useful tool, both experimentally and computationally, for distinguishing and defining functional zones (Hutcheon and Yarom, 2000; Izhikevich et al., 2003; Nusser, 2009; Zhuchkova et al., 2013). That is, non-uniform ionic conductance expression throughout the neuronal membrane can establish distinct regions that differ in terms of their frequency preference to subthreshold oscillatory input. Therefore, in biologicallyrealistic compartmental models of pyramidal neurons, the particular ionic conductances that are defined for a given model compartment largely determines the resonant properties of that compartment (Reyes, 2001; Lörincz et al., 2002). Differences in compartment resonant properties can, in turn, be interpreted in terms of different functional roles. Hu et al. (2009), used resonance analysis on CA1 hippocampal pyramidal neuron models (as well as experimentally) and identified two complementary resonances (roughly 3–12 Hz), each generated by distinct mechanisms, for signals transmitted to-and-from the soma and distal apical dendrite when membrane potentials are below −55 mV.

In the current study, we employ a resonance-based computational approach to studying the spatial distribution of frequency preference within a realistic layer 5 pyramidal neuron model. We began with a model whose post-synaptic responses have been tuned to generate action potentials in response to simultaneous input to distal apical regions and the soma to simulate the associative function of these neurons. We examined the resonance responses of this model to oscillatory input in biologically relevant frequency ranges (1–40 Hz). We then used transfer resonance to examine the electrical coupling that occurs between distinct regions of the model via both subthreshold oscillations and the generation of both perisomatic and distal apical action potentials. We find that tuning of a neuron's post-synaptic physiological properties to enhance association between feedforward and feedback inputs impacts which frequencies it is expected to respond to most strongly and reinforce.

### MODELS AND ANALYSIS

#### Model Configuration

We used the GEneral NEural SImulation System **(**GENESIS) environment to construct a model neuron that has properties of neocortical layer 5 pyramidal neurons (Bower and Beeman, 2003). Our model is an adaptation of the regular spiking, tufted layer 5 neuron constructed by Traub et al. (2005) (for morphology, passive electrical properties) and Traub et al. (2003) (for conductance kinetics). We made no changes to morphology. All but two of the ionic channels included in the original model remain throughout our simulations. Specifically, resonance analysis requires that neuronal responses be subthreshold. To assure subthreshold behavior in our resonance simulations, we therefore removed the fast sodium conductance [gNa(F)], which produces action potential onset (Hodgkin and Huxley, 1952) and the Ca2<sup>+</sup> dependent K<sup>+</sup> afterhyperpolarization current which acts as an integrating current to control bursting behavior (Mainen and Sejnowski, 1996). Furthermore, the kinetics for all but one of the ionic conductances remained the same. The lone adjustment was made to M current [gK(M)] kinetics where we shifted activation dynamics up to 15 mV in the hyperpolarized direction near the base of the activation curve, and reducing its time constant maximum value by 40 ms (see Equations S1, S2 and Figure S1). Changing gK(M) in this way allows this channel to become active at subthreshold membrane potentials, which more accurately captures the resonance characteristics attributed to gK(M) in neocortical pyramidal neurons when their membrane potential is between rest and depolarized to 30 mV (Gutfreund et al., 1995; Hutcheon and Yarom, 2000). In addition we updated some conductance density values which have been shown experimentally to diverge from those in the original (Traub et al., 2005) model. A more complete description of the model configuration is found in the Supplementary Materials.

Our model was configured to behave similarly to layer 5b pyramidal neurons that display back propagation-activated Ca2<sup>+</sup> spikes in distal apical dendrite (Larkum et al., 1999; Hay et al., 2011; Larkum, 2013). We incorporate the concept of a distal apical "hot zone," which is defined as distal apical dendrite (model compartments 16–18 in Table S1) and apical tuft (model compartments 19) with densities for the high-threshold Ca2<sup>+</sup> conductance [gCa(H)] and low-threshold Ca2<sup>+</sup> conductance [gCa(L)] 10 and 100 times higher than anywhere else in the apical dendrite, respectively. This type of configuration has been shown to faithfully reproduce the distal apical dynamics, such as Ca2<sup>+</sup> spike generation, that are necessary for the coupling of perisomatic and distal apical regions of layer 5b pyramidal neurons (Hay et al., 2011).

#### Model Behavior

In order to determine if our model reproduced feedforwardfeedback interactions, we injected a depolarizing current into the soma and distal apical dendrite compartments (**Figure 1A**) with gNa(F) active. A 3 nA, 5 ms square pulse injected into the soma of the model with the hot zone (**Figure 1B**, leftbottom) evoked an action potential spike and spikelet followed by an afterdepolarization potential (**Figure 1B**, left-top, black line). This somatic depolarization back propagates to distal apical dendrite and leads to depolarization of the local membrane potential (**Figure 1B**, left-top, green line). When a back propagated pulse evoked by a somatic square wave injection arrives at distal apical dendrite at the same time as a current injection (given by equation S3) into the distal apical dendrite (**Figure 1B**, right-bottom), a very broad Ca2<sup>+</sup> spike can be produced in a spike initiation region in the distal apical dendrite (**Figure 1B**, right-top, green line). This Ca2<sup>+</sup> spike travels along the apical dendrite to the soma and can cause the soma to depolarize further and emit an additional action potential (**Figure 1B**, right-top).

**Figure 1C** illustrates the scenario when the distal apical hot zone has been removed, such that compartments comprising distal apical dendrite (16–18, see Table S1) and apical tuft (19) were given the same density values for gCa(H) and gCa(L) as the rest of the apical dendrite (6–15). A 3 nA, 5 ms square pulse injected into soma (**Figure 1C**, left-bottom) evoked an action potential spike and spikelet followed by an afterdepolarization potential (**Figure 1C**, left-top, black line) that was reduced in duration relative to the neuron containing the hot zone (**Figure 1B**, lefttop, black line). The somatic depolarization back propagates to distal apical dendrite and leads to a depolarization of the local membrane potential (**Figure 1C**, left-top, green line) that is also reduced relative to the neuron containing the hot zone (**Figure 1B**, left-top, green line). When a back propagated pulse evoked by a somatic square wave injection arrives at distal apical dendrite at the same time as a current injection into the distal apical dendrite (**Figure 1C**, right-bottom), a less robust Ca2<sup>+</sup> spike is initiated in the distal apical dendrite, indicating a reduction in electrical coupling between distal apical dendrite and soma. The result is that the soma does not depolarize further and does not emit an additional action potential as it did in the case with the distal apical hot zone (compare **Figures 1C,B**, right-top, black lines).

#### Resonance Analysis

Resonance analysis was applied on both the model neuron in **Figure 1B**, the one with a distal apical hot zone, and the model neuron in **Figure 1C**, the one without a hot zone. In both cases, gNa(F) was removed to prevent action potential generation, as described in section Model Configuration, above. Toggling the hot zone in this manner allowed resonance analysis to be used as a tool to investigate the role of gCa(H) and gCa(L) in the coupling of distal apical and perisomatic regions of the model. There were two phases to our resonance analysis: input and transfer.

#### Input Resonance Analysis

Input resonance analysis was used to examine local frequency preference throughout the model. To characterize the input resonance properties of our model neuron, we systematically injected "chirp" currents, or, broad-spectrum sinusoidal currents with linearly increasing frequency (0–40 Hz over 65 s) into a compartment within the following regions of the model neuron: soma (specifically, compartment 2 in Table S1), basal dendrite (5), middle apical dendrite (11), distal apical dendrite (18), and apical tuft (19) (**Figures 2A,B**). 40 Hz was set as the maximum frequency for the analysis based on preliminary work where the soma and distal apical dendrite compartments were held at depolarized membrane potentials and injected with a chirp current with linearly increasing frequency from 0 to 200 Hz over 325 s. During these tests, resonance was not observed above 30 Hz, therefore, we report model behavior up to 40 Hz in order to fully capture all resonance observed.

During the input resonance analysis, compartments' membrane potentials were varied from −80 to −30 mV for soma and basal dendrite, and −80 to 0 mV for apical dendrite and apical tuft compartments, in steps of 5 mV using a DC offset current (for DC offset values see Supplementary Materials). These membrane potential ranges were selected based on physiological considerations to avoid evaluating membrane resonance during significant refractory periods. For instance, the maximum membrane potential for the soma and basal dendrite compartments was selected based on the peak value of the somatic afterdepolarization potential after the neuron has fired an action potential; the maximum membrane potential for apical dendrite and tuft compartments were selected based on peak membrane potential after local current injection is timed to match the arrival of a back propagated somatic action potential (see **Figure 1B**, right-top).

At each of these membrane potentials, a chirp current was injected into the compartments and we calculated the input impedance by dividing the Fourier spectrum of the resultant membrane potential by the Fourier spectrum of the chirp current, FVm and F<sup>I</sup> , respectively (Equation 1). Chirp current amplitudes were chosen that kept the resultant membrane potential oscillations below approximately 8 mV in peak-to-peak amplitude for each compartment. This ensures that sufficiently non-overlapping regions of the activation curves for the ionic conductances in the model are examined. The chirp current amplitudes were as follows: 10 pA, basal dendrite; 50 pA, apical tuft; 75 pA, soma; 100 pA, middle apical dendrite; and 115 pA, distal apical dendrite. We obtained our Fourier spectrums by applying a Fast Fourier Transform (FFT) algorithm (MatlabTM FFT function) on model compartment's membrane potential over the 65 s chirp injection epoch and applying 100-pt moving

and EPSP-like current injection to distal apical dendrite (1 nA peak, 5 ms delay relative to somatic step current) without hot zone.

average smoothing in the frequency domain. Impedance was calculated on the interval, 1–40 Hz, to avoid boundary effects associated with applying discrete FFT on a finite sampling window. Resonance quantification analysis was performed on the resultant impedance magnitude curves (simply referred to as impedance in the remainder of the text) by calculating resonance strength (Q), degree of high-pass filtering (D), half-band width (HB), and by identifying the resonant frequency (fres) and resonant impedance (Zres) value (Erchova et al., 2004) (Equations 2, 3; Figure S8).

We introduce a dimensionless quantity that we call "resonance quality" defined as the ratio of resonance strength and degree of high-pass filtering (Q/D). We used resonance quality as the primary metric to quantify the shape of impedance curves within the 1–40 Hz interval (Equation 4). It is advantageous to use Q/D to mitigate "false positive" resonance cases when Q is roughly equal to D (Q/D ≈ 1), a condition that is more indicative of broadband or high-pass filtering/amplification as opposed to a well-defined resonance. On the other hand, larger values of Q/D are indicative of well-defined band-pass filtering (large amplitude and narrow peak in impedance curve). Mitigating for false positives becomes important when a compartment passes through its resonant regime as its membrane becomes increasingly depolarized. It was observed during preliminary tests that resonance tends to give way to high-pass filtering in some simulations, particularly for the perisomatic regions of the model. Therefore, Q/D is a metric that accurately captures resonance features of our model on the 1–40 Hz interval. Expressions for our resonance quantification analysis are given in Equations 1–4:

$$Z = \frac{F\_{Vm}}{F\_I} \tag{1}$$

$$Q = \frac{Z\_{\text{res}}}{Z\_1} \tag{2}$$

$$D = \left. \frac{Z\_{40}}{Z\_1} \right. \tag{3}$$

$$\text{resonance quality} = \frac{Q}{D} = \frac{Z\_{\text{res}}}{Z\_{40}} \tag{4}$$

where Z, FVm, and F<sup>I</sup> are the impedance, Fourier spectrum of output compartment membrane potential, and Fourier spectrum of injected chirp current, respectively; Zres, Z1, and Z<sup>40</sup> are the resonant impedance, impedance at 1 Hz, and impedance at 40 Hz, respectively (**Figure 2C**).

#### Transfer Resonance Analysis

Transfer resonance analysis was used to characterize subthreshold interaction, or electrical coupling, between compartments in our model. Transfer impedance was calculated between each dendritic compartment examined in the input resonance analysis and the soma, as well as the transfer impedance between the soma and distal apical dendrite. Transfer impedance was calculated by dividing the Fourier spectrum of the resultant membrane potential in the transfer, or "receiving," compartment by the Fourier spectrum of the chirp current injected into the input compartment. Transfer resonance was quantified using the same metrics used in the input resonance analysis. The compartments where the chirp current was injected had their membrane potentials varied across the same range as in the input resonance analysis, but in steps of 10 mV, as opposed to 5 mV, using DC offsets (for DC offset values see Supplementary Materials). In addition, the transfer analysis was repeated multiple times, each time with the transfer compartments held at either hyperpolarized, near rest, or various amounts of depolarized membrane potentials using a second DC offset (see Supplementary Materials).

#### RESULTS

### Input Resonance Analysis on Model With Hot Zone

Results of the input resonance analysis are shown in **Figure 3** (for waveforms of compartment voltage response at key values of membrane potential in different regions of the model neuron, see Figure S3). Profiles of input resonance quality (Q/D) are presented in **Figure 3A**. For the compartments along the soma-apical dendrite axis, there was a progressive shift in the compartment exhibiting the highest resonance quality as each compartment's membrane potential was increased from hyperpolarized to depolarized potentials. The soma's peak resonance quality, the highest of any compartment, occurred at hyperpolarized membrane potentials. Also note that the soma's resonance quality (**Figure 3**, black line) reduced to approximately 1 at a membrane potential of around −35 mV; a condition that is not indicative of resonance, but rather of a broadband and/or high-pass impedance curve. Middle apical dendrite resonance quality (blue line) peaked when its membrane potential reached around −65 mV, and it became the compartment with the highest resonance quality. Distal apical dendrite (green line) became the compartment with the highest resonance quality at more depolarized membrane potentials, with a peak at −45 mV, approximately 12 mV above the local resting membrane potential. By contrast, peak resonance quality for apical tuft (red line) occurred around −20 mV, but there was no membrane potential for which apical tuft became the compartment with the highest resonance quality. Basal dendrite (purple line) exhibited only very weak resonance for a small range of membrane potentials, from hyperpolarized potentials up to approximately −55 mV.

In addition to a soma-apical dendrite shift in peak input resonance quality as each compartment's membrane potential was increased from hyperpolarized to depolarized

membrane potentials, the input resonance quality curves for the compartments on the soma-apical dendrite axis, including the apical tuft, became progressively flatter the farther the compartment was from the soma. In particular, apical tuft displayed a fairly flat resonance quality curve across almost the entire range of membrane potentials. This progressive flattening of input resonance quality profiles along the soma-apical dendrite axis is similar to the spatial distribution of frequency preference specificity as reflected in the half-bandwidth (HB) of the compartments' impedance curves (Table S2A). The soma and middle apical dendrite have narrow HB at hyperpolarized and resting membrane potentials but their HB widens rapidly with depolarization. On the other hand, distal apical dendrite and apical tuft generally have wider HB than soma or middle apical dendrite. However, the HB of these compartments does become narrower for intermediate levels of depolarization (by up to 9 Hz in some cases for distal apical dendrite).

Input resonant frequency profiles for the compartments in this analysis are shown in **Figure 3B.** Soma, basal dendrite, and middle apical dendrite all consistently showed rising resonant frequency as membrane potential increased from hyperpolarized to higher levels of depolarization. Somatic resonant frequency increased from 5 Hz at hyperpolarized potentials to 23.5 Hz at −40 mV, the highest membrane potential for which the soma exhibited resonance. Basal dendrite exhibited resonance across an even shorter range of membrane potentials where it sweeps out a wide range of resonant frequencies from 8.5 to 29 Hz. Middle apical dendrite had a resonant frequency profile similar to the soma but with a slower rate of increase. Distal apical dendrite and apical tuft had relatively flat resonant frequency curves, centered around 8–9.5 Hz from hyperpolarized to near resting membrane potentials. The resonant frequency curves for these compartments ultimately increased to > 15 Hz for highly depolarized membrane potentials. Note that at −50 mV, distal apical dendrite began to display a resonant frequency that progressively increased with increasing membrane potentials, much like the soma, basal dendrite, and middle apical dendrite compartments. On the other hand, apical tuft remained relatively flat out to −20 mV, then rapidly increased with further membrane depolarization.

Input resonant impedance profiles for this analysis are shown in **Figure 3C**. The compartment that showed the highest resonant impedance values was the basal dendrite. Its resonant impedance was 700 M when its membrane potential was −80 mV and rapidly decreased to 500 M as its membrane potential increased to −55 mV. The maximum resonant impedance values for the apical dendrite and apical tuft compartments showed an inverse relationship to these compartments' maximum values in their resonance quality profiles, namely, the highest resonant impedance values belonged to the apical tuft compartment across the entire range of membrane potentials (maximum value 150 M at −45 mV), while the distal apical dendrite was generally greater than middle apical dendrite, except below −70 mV. Distal apical dendrite resonant impedance maximum was 60 M at −45 mV while middle apical dendrite resonant impedance maximum was 37 M at −65 mV. The soma's resonant impedance profile peaked at hyperpolarized potentials (50 M at −75 mV) then decreased as membrane potentials increased to −40 mV.

# Transfer Resonance Analysis on Model With Hot Zone

#### Soma and Distal Apical Compartments

Transfer resonance analysis was used to examine subthreshold electrical coupling between the dendritic compartments involved in the input resonance analysis and the soma, and between the soma and distal apical dendrite. Because our baseline model neuron is configured to exhibit enhanced spiking due to coupling of the soma and a distal apical spiking zone, we chose to present the results of transfer resonance analysis for the soma and the two distal apical compartments together (**Figure 4**, Figure S4). The black curves represent the scenarios in which a chirp current was injected into the soma and the resultant membrane potential in the distal apical dendrite was measured. The green curves represent the scenarios in which a chirp current was injected into distal apical dendrite and the resultant membrane potential of the soma was measured. The red curves represent the scenarios in which a chirp current was injected into apical tuft and the resultant membrane potential of the soma was measured. Multiple experimental runs were performed in which the injection compartment membrane potential was stepped through a range of values in steps of 10 mV while the transfer compartment was held at a few select membrane potentials provided in parentheses next to the name of the injection compartment in the legend of **Figure 4**.

**Figure 4A** presents the transfer resonance quality profiles for the soma and distal apical compartments. The somatic membrane potentials that resulted in the highest resonance quality between soma and distal apical dendrite were hyperpolarized/resting potentials. As the soma was depolarized above its resting membrane potential, transfer resonance quality decreased rapidly. The highest observed values of resonance quality between soma and distal apical dendrite (Q/D between 4.5 and 5) occurred both when the distal apical dendrite membrane potential was near rest and depolarized to −30 mV. The resonance quality profile for soma-to-distal apical dendrite transmission remains qualitatively the same but is shifted downward when distal apical dendrite membrane potential is hyperpolarized to −75 mV (values between 3.5 and 4 when soma at hyperpolarized/resting membrane potentials) or depolarized further to −3 mV (values between 2.5 and 3 when soma at hyperpolarized/resting membrane potentials).

The membrane potentials of distal apical dendrite that resulted in the highest transfer resonance quality with the soma were between −50 and −40 mV (**Figure 4A**). As distal apical dendrite membrane potential became more hyperpolarized or more depolarized than these values, transfer resonance quality decreased. The highest observed values of transfer resonance quality between distal apical dendrite and soma (Q/D approximately 5.5) occurred when the somatic membrane potential was hyperpolarized at −80 mV. The resonance quality profile for distal apical dendrite-to-soma transmissions qualitatively remains the same, but is shifted downward when the soma becomes more and more depolarized—peak values between 4.5 and 5 when soma at −65 mV, and peak values between 2.5 and 3 when soma at −50 mV.

The largest transfer resonance quality observed for signals transmitted from apical tuft to soma also occurred when the soma was hyperpolarized and at rest (**Figure 4A**). Each of these profiles contain values > 4, and their peaks (5.2 and 4.8, respectively) occurred when the apical tuft compartment was at −20 mV. The peak value in the profile for signals transmitted from apical tuft to soma when soma is depolarized to −50 mV reduced to 2.8.

**Figure 4B** shows the transfer resonant frequency between the soma and distal apical compartments. Transfer resonant frequency from soma to distal apical dendrite increases almost linearly as soma membrane potential is depolarized. When distal apical dendrite membrane potential is between −75 and −30 mV, the resonant frequency profiles of soma-to-distal apical dendrite transmission group together and vary from 7 Hz for hyperpolarized soma (distal apical dendrite at −75 or −60 mV) to 18 Hz for soma depolarized to −30 mV (distal apical dendrite at −30 mV). The highest resonant frequencies for transmission from soma to distal apical dendrite occurred when distal apical dendrite is highly depolarized to −3 mV. Under this condition, the range of somatic transfer resonant frequencies as the soma's membrane potential was varied is 11–34 Hz. However, it should be noted that the frequencies under the condition of both high somatic and distal apical dendrite depolarization are associated with very low resonance quality and therefore represent a very weak frequency preference.

Transfer resonant frequency from distal apical dendrite to soma displayed a banded structure that is based on somatic membrane potential (**Figure 4B**). The band with the slowest frequencies (roughly 7–11 Hz) occurred when the soma was hyperpolarized; a band with intermediate frequency values (10– 14 Hz) occurred when soma was at −65 mV; lastly, a band with the fastest frequencies (14–20 Hz) occurred when soma was depolarized to −50 mV. On the other hand, each of the transfer resonant frequency curves for apical tuft-to-soma transmissions were flatter and non-overlapping. These bands were situated at 7, 10, and 14.5 Hz for hyperpolarized, resting, and depolarized soma, respectively. The bands in resonant frequency for transmission from distal apical dendrite and apical tuft to soma are inversely related to the resonance quality of transmission from distal apical dendrite and apical tuft to the soma, such that the faster the frequency, the lower the transfer resonance quality.

Transfer resonant impedance between soma and distal apical compartments is shown in **Figure 4C**. Transfer resonant impedance profiles for soma-to-distal apical dendrite transmission and for transmission between both distal apical compartments and the soma are qualitatively similar to their corresponding resonance quality profiles. However, it is easier to identify within the transfer resonant impedance profiles a two-tiered structure in the communication between these two regions of the pyramidal neuron model. For soma-to-distal apical dendrite transmission, the transfer resonant impedance profiles for the cases when distal apical dendrite is at −60 and −30 mV group together, peaking between 13 and 14 M when

soma is hyperpolarized, and decreasing rapidly to 0.5 M with somatic depolarization. On the other hand, the transfer resonant impedance profiles for the case when distal apical dendrite membrane potential is hyperpolarized at −75 mV and depolarized strongly to −3 mV group together, with values around 6 M when soma is hyperpolarized and also decreasing to 0.5 M with somatic depolarization.

For distal apical dendrite-to-soma transmission, the transfer resonant impedance profiles for the cases when the soma is at −80 and −65 mV group together, peaking near 15 M when distal apical dendrite is between −50 and −40 mV, and decreasing rapidly to 5 M with either hyperpolarization or depolarization out of this range of dendritic membrane potentials. On the other hand, when the soma is depolarized to −50 mV, the peak in the transfer resonant impedance profile for distal apical dendrite-to-soma transmission is reduced by half to 7.5 M and tapers to 3 M with either hyperpolarization or depolarization out of the −50 to −40 mV range in membrane potential values for distal apical dendrite.

The transfer resonant impedance profiles for apical tuftto-soma transmissions were generally lower in magnitude, except for compartment membrane potentials > −25 mV, and had a broader shape relative to the profiles for distal apical dendrite. Maximum values for apical tuft-to-soma transmission were 12 M when soma was either hyperpolarized or at rest, while maximum values of the apical tuft-to-soma transfer resonant impedance profile decreased to 6 M when soma was depolarized.

#### Basal Dendrite and Middle Apical Dendrite

Transfer resonance analysis for the other dendritic compartments included in this study are presented in **Figure 5** and Figure S4. Each curve was obtained by injecting chirp current into the respective compartment and then measuring the resultant membrane potential in the soma. Once again, there are three profiles per compartment corresponding to the three membrane potential values that the transfer compartment, in this case the soma, was held at–hyperpolarized, resting, and depolarized potentials.

**Figure 5A** presents the transfer resonance quality profiles for middle apical and basal dendrite compartments (blue and purple lines, respectively). The largest transfer resonance quality observed were for signals transmitted from middle apical dendrite to soma when soma was at hyperpolarized and resting membrane potentials. These values peaked (3.7 and 3.1, respectively) when middle apical dendrite membrane potential was between −70 and −55 mV and then reduce sharply (to 1.5) as the compartment is depolarized. The resonance quality profile for transmission between middle apical dendrite and soma when the soma is depolarized to −50 mV exhibits the typical downward shift relative to the profiles obtained when soma was at hyperpolarized/resting membrane potentials. In general, the lowest transfer resonance quality was observed to

be in transmissions from basal dendrite to soma. These profiles exhibited the usual banding structure where the profiles obtained when soma was at hyperpolarized and resting potentials had the higher values of transfer resonance quality and were clustered much closer together. The transfer resonance quality between basal dendrite and soma peaked at 2.8 when both compartments were hyperpolarized, and when the soma was depolarized to −50 mV, transfer resonance between basal dendrite and soma nearly disappears.

Transfer resonant frequency for middle apical and basal dendrite compartments is shown in **Figure 5B**. As soma membrane potential was increased from hyperpolarized to depolarized values, the transfer resonant frequency became progressively faster for signals transmitted from both middle apical and basal dendritic compartments to the soma. The bands of resonant transfer frequency for middle apical dendrite-tosoma and basal dendrite-to-soma transmissions increased as dendritic membrane potential increased and were partially overlapping. For hyperpolarized, resting, and depolarized soma, the corresponding middle apical dendrite-to-soma bands were 5–15 Hz, 10–20 Hz, and 15–25 Hz, respectively. Similarly, for hyperpolarized, resting, and depolarized soma, the corresponding basal dendrite-to-soma bands were 5–11 Hz, 10–14 Hz, and 17–21 Hz.

Transfer resonant impedance for middle apical and basal dendrites is shown in **Figure 5C.** Like both distal apical compartments, basal dendrite and middle apical dendrite compartments had transfer resonant impedance profiles for the cases when the soma was at −80 and −65 mV that group together at much higher values than when the soma was at −50 mV. For basal dendrite-to-soma transmissions, the highest transfer resonant impedance values (40–45 M) are observed for hyperpolarized basal dendrite compartment and hyperpolarized and resting soma. When basal dendrite is depolarized, the transfer resonant impedance decreases rapidly to values near 5 M. When the soma is depolarized to −50 mV, the maximum transfer resonant impedance is 22 M at hyperpolarized basal dendrite membrane potential and decreases to 3 M when basal dendrite was depolarized to −30 mV. For middle apical dendrite-to-soma transmissions, the highest transfer resonant impedance values (25–28 M) are observed for middle apical dendrite membrane potentials between −70 and −50 mV and hyperpolarized and resting soma. When middle apical dendrite is hyperpolarized or depolarized out of this range, the transfer resonant impedance decreases. When the soma is depolarized to −50 mV, the maximum transfer resonant impedance is 15 M at hyperpolarized dendrite membrane potential and decreases to 3 M when middle apical dendrite is depolarized to 0 mV.

#### Resonance Analysis on Model Without Hot Zone

#### Input Resonance Analysis

A good illustration of changes to model behavior when the distal apical hot zone has been removed is to examine difference plots between the case with the hot zone and the case without the hot zone. **Figure 6** illustrates how the outcome of our input

resonance analysis changed when gCa(H) and gCa(L) density in distal apical dendrite and apical tuft is reduced. For profiles of input resonance quality, resonant frequency, and resonant impedance presented in the same format as **Figure 3**, and for the corresponding waveforms of compartment voltage response at key values of membrane potential, see Figures S5, S6, respectively.

Percentage change to input resonance quality when distal apical hot zone has been removed is shown in **Figure 6A**. Not surprisingly, there isn't much change to the resonance quality in the basal dendrite compartment. There was a small decrease in resonance quality for soma and middle apical dendrite at hyperpolarized/rest membrane potentials (1–2.5% decrease). On the other hand, much larger decreases to resonance quality was observed for the distal apical compartments where the gCa(H) and gCa(L) density was reduced. Two large decreases occur at membrane potentials of −70 and −20 mV. At −70 mV, resonance quality decreased by 6 and 14% for apical tuft and distal apical dendrite compartments, respectively. At −20 mV, resonance quality decreased by 7 and 23% for apical tuft and distal apical dendrite compartments, respectively.

Reducing gCa(H) and gCa(L) density in distal apical compartments also impacts HB values for the compartments during input resonance analysis (Table S2B). The soma and distal apical compartments all experienced a widening of HB relative to the results obtained on the baseline model, while middle apical dendrite and basal dendrite both experienced a modest narrowing of HB.

**Figure 6B** shows how each compartments' input resonant frequency changed when the hot zone was removed. Resonant frequency for the basal dendrite compartment does not change when distal gCa(H) and gCa(L) density is reduced. There is a small decrease to resonant frequency ( < 1 Hz) for soma and middle apical dendrite when they are at hyperpolarized membrane potentials. Larger decreases to resonant frequency are observed for distal apical compartments at hyperpolarized and very high levels of depolarization. Both distal apical dendrite and apical tuft experience up to 2 Hz reduction to resonant frequency at −65 mV, while at membrane potentials > −20 mV, the reduction experienced by distal apical dendrite and apical tuft is as much as 8 and 3.8 Hz, respectively.

Changes to input resonant impedance is shown in **Figure 6C**. With the reduction to distal gCa(H) and gCa(L) density, resonant impedance for the basal dendrite compartment increased by 1 M at hyperpolarized membrane potentials. There was a small decrease to resonant impedance for soma and middle apical dendrite when they were at hyperpolarized membrane potentials ( < 1 M). Like the case for resonance quality, the distal apical dendrite and apical tuft compartments show significant reductions in resonant impedance at −70 and −20 mV except this time, it is the apical tuft that was most affected by the reduced distal gCa(H) and gCa(L) density. At −70 mV, resonant impedance magnitude decreased by 6 and 13 M for distal apical dendrite and apical tuft compartments, respectively. At −20 mV, resonant impedance magnitude decreased by 13 and 24 M for distal apical dendrite and apical tuft compartments, respectively.

#### Transfer Resonance Analysis

In **Figure 7** we again use difference plots to illustrate changes to model behavior when the distal apical hot zone has been removed. Profiles of transfer resonance quality, resonant frequency, and resonant impedance are presented in the same format as **Figures 4**, **5**, and for the corresponding waveforms of compartment voltage response at key values of membrane potential, see Figures S7–S9, respectively. In the case of transfer resonance analysis performed on a model neuron without a distal apical hot zone, only simulations with the transfer compartment held near resting membrane potentials were performed (−60 mV for distal apical dendrite and −65 mV for soma).

**Figure 7A** presents the percentage change to transfer resonance quality. In general, when distal gCa(H) and gCa(L) density is reduced, there is a decrease in the transfer resonance quality between soma and distal apical dendrite, as well as between the dendritic compartments and the soma. Notably, the somatic transfer resonance quality decreased almost uniformly by 7–8%. In addition, the transfer resonance quality between distal apical dendrite and soma experienced reduced values at −70 mV (17% reduction) and −20 mV (22% reduction). In addition, the impact of removing the distal hot zone on HB values in the case of transfer resonance is not as clear as in the case of input resonance (Table S2D).

The change to transfer resonant frequency and transfer resonant impedance is presented in **Figures 7B,C**. For all compartments, both transfer resonant frequencies and transfer resonant impedance magnitudes changed very modestly in response to reduced distal gCa(H) and gCa(L) density. In the case of transfer resonant frequencies, the changes were no more than ±1.5 Hz. In the transfer resonant impedance profiles, the distal apical compartments showed reduced values at the same membrane potentials (−70 and −20 mV) at which reductions were observed in their input and transfer resonance quality, as well as input resonant impedance. In particular, the transfer resonant impedance of distal apical dendrite and apical tuft (as well as the soma) decreased by 1–2 M when the compartment membrane potential was −70 mV. When membrane potential was −20 mV, transfer resonant impedance for distal apical dendrite and apical tuft decreased by 2–3 M.

### DISCUSSION

Using the concept of a distal "hot zone," we have configured a model neocortical layer 5 pyramidal neuron to display enhanced coupling of perisomatic and distal apical spiking zones (Hay et al., 2011). Using this model, we applied both input and transfer resonance analysis in the soma and in several key locations within the dendritic tree to assess functionally-relevant differences in the response of different zones of the model. Furthermore, to gain insight about the role of distal gCa(H) and gCa(L) density in the coupling of perisomatic and distal apical regions, we performed the resonance analysis twice, once with the distal hot zone and once without.

# General Pattern in Resonance Characteristics

At least four distinct regions of the model neuron can be distinguished based on the results of our resonance analysis. In describing these zones, we use the following convention for defining relevant brainwave bands (Buzsaki, 2011): theta (4– 7 Hz), alpha (7–14 Hz), and beta (14–30 Hz).

Basal dendrite (and oblique dendrite, based on previous work not included in the current study) is very weakly resonant only at subthreshold membrane potentials and has input resonant frequencies that increase rapidly throughout alpha and beta range as membrane potential increases. It is more accurate to consider these compartments as having a very weak frequency preference spectrum at subthreshold membrane potentials and transitioning to a high-pass filter at suprathreshold potentials. Basal dendrite also had much higher input and transfer resonant impedance at near rest/hyperpolarized membrane potentials than any other compartment in the analysis. The high input and transfer impedance is partially due to the small size of this compartment and its close proximity to the soma, respectively.

The apical tuft has weak input resonance across the entire range of membrane potentials considered and has a flat input resonant frequency profile that is almost entirely confined to alpha frequencies. On the other hand, apical tuft has very strong transfer resonance with a flat transfer resonant frequency profile that varies depending on soma membrane potential—low alpha when soma hyperpolarized, high alpha when soma depolarized, in our simulations. The apical tuft had moderately high values of input and transfer resonant impedance across a very broad range of membrane potentials, particularly when membrane potential was > −50 mV.

The soma and middle apical dendrite can be said to belong to a perisomatic zone along the soma-apical dendrite axis. These compartments have very strong input resonance at hyperpolarized/resting membrane potentials, and their resonant frequency profiles increase from theta to mid beta as compartment membrane potential is increased. Both compartments had low overall input resonant impedance, but moderate to high transfer resonant impedance at resting and hyperpolarized membrane potentials.

Distal apical dendrite showed strong input resonance at moderate levels of depolarization, and a relatively flat input resonant frequency profile in low alpha range when membrane potential was < −50 mV, but an increasing profile from low alpha to mid beta when membrane potential was > −50 mV. Like apical tuft, distal apical dendrite also showed very strong transfer resonance with a relatively flat transfer resonant frequency profile that varies depending on soma membrane potential—theta/alpha border when soma hyperpolarized, alpha/beta border when soma depolarized. Distal apical dendrite had moderate values of resonant input and transfer impedance and the peak in both

of these profiles occurs at depolarized membrane potentials approximately −40 mV.

Because our model was configured to have a distal apical hot zone where gCa(H) and gCa(L) density is higher than the rest of the model, it is not surprising that resonance analysis is able to distinguish between perisomatic regions and distal apical regions. However, our results show that resonance analysis is also able to further divide the distal apical hot zone into two distinct zones, namely, distal apical dendrite and apical tuft. Our results indicate that the frequency preference of these distal regions of layer 5 pyramidal neurons becomes more specific at moderate to high levels of depolarization as evidenced by their input and transfer resonance quality scores, and their HB values. This increase in frequency preference is directly correlated to the impact of these distal apical regions on the soma as evidenced by their transfer resonant impedance. On the other hand, our results indicate that the frequency preference of proximal apical dendrite and soma is strongest at near resting and hyperpolarized membrane potentials (high input and transfer resonance quality, and small HB) and that, at least in the case of the soma, this is the condition under which signals transmitted to distal apical regions will have the highest transfer resonant impedance, and therefore, the largest functional impact on action potential generation. In short, the shifts along the soma-apical dendrite axis in maximum input resonance quality and flatness of input resonance quality profiles indicates that the further a compartment is from the soma on this axis, the weaker the frequency preference that it can attain, the more depolarized it needs to be to attain its peak frequency preference, and the weaker its dependence on a particular range of membrane potentials for resonance to occur. Furthermore, our results indicate that the basal dendrite (and presumably oblique dendrites) of layer 5 pyramidal neurons do not have strong frequency preferences. Their close proximity to the soma may not require the same level of specificity in frequency preference to have a large impact on the soma.

The four distinct zones identified in the current study are consistent with a three-layer model of layer 5 pyramidal neuron presented by Spruston and Kath (2004), which is based on synaptic integration and afferent connectivity throughout the neuronal membrane. The first layer of this model is an input layer comprising two distinct zones—perisomatic dendrites (basal and oblique), and apical tuft. The output of the first layer feeds into a second layer that acts as an integration layer. The second layer is comprised of two distinct integration zones—proximal apical dendrite and soma, and distal apical dendrite. The third layer is the action potential initiation zone in the axon hillock. The four zones identified in our resonance analysis correspond to the four zones used by Spruston and Kath (2004) to define the first two layers of their model layer 5 pyramidal neuron. In this view, layer 5 pyramidal neurons can be considered as having two input zones with weak input frequency preference, one close to the soma comprising basal and oblique dendrites, and one far from the soma in the apical tuft. However, significant frequency preference in signals transmitted from both of these input zones to the soma emerges due to the rather strong frequency preference displayed by the two integration zones of the neuron—distal apical dendrite and proximal apical dendrite/soma.

# Conditions for Optimum Subthreshold Communication Between Soma and Distal Apical Dendrite

Transfer resonance analysis also provides an indication of the conditions that should be most conducive to electrical coupling between the subthreshold perisomatic zone and the distal apical dendrite zone in pyramidal neurons. Communication between these two regions should be best when the soma is at hyperpolarized/resting membrane potentials while distal apical dendrite is depolarized to −40 mV and the signal encoding the information has a frequency of 7–12 Hz. Because of the shift in maximum input resonance quality along the soma-apical dendrite axis, the conditions for maximum electrical coupling between soma and distal apical dendrite also coincides with conditions for maximum frequency preference (input resonance quality) and response strength (input resonant impedance) to injected chirp currents for these two regions. In addition, there are key membrane potential values for which transfer resonant frequencies for some or all compartments converge onto the same value. For example, when the soma and distal apical regions are both hyperpolarized to −80 mV, the resonant transfer frequency for communication in both directions is 7.5 Hz (low alpha); when all compartments are near the resting membrane potential of the soma, −68 mV, they all have a transfer resonant frequency around 10 Hz (middle alpha); on the other hand, when the soma and distal apical regions are at −55 mV, they have a transfer resonant frequency of 15 Hz (low beta). It has been suggested that global coherence within dendritic oscillators plays a major role in the modulation of perisomatic spike generation (Remme et al., 2009). Our results suggest that a more homogeneous distributions of critical membrane potential values throughout the different regions of layer 5 pyramidal neurons may be associated with global coherence within these neurons.

It should be noted that when a DC offset is used to bring distal apical dendrite membrane potential to −40 mV, the somatic compartment remains near rest because it experiences very little of this distally-applied current. Therefore, situations when a layer 5 pyramidal neuron is at or below rest when it experiences input to its distal apical regions is sufficient to create the conditions for optimum modulation of somatic membrane potentials. Such a situation can arise when a quiescent period for a region of neocortex gives way to increased stimulation via input to distal regions of layer 5 pyramidal neurons, such as non-specific thalamocortical input or corticocortical feedback from distant neocortical areas (Spruston, 2008). If the input to the distal apical dendrite and apical tuft raises the local membrane potential to an average value of −40 mV while encoding an information signal at 7–12 Hz, it would maximize the subthreshold electrical coupling of distal apical dendrite and soma. During such conditions, oscillations in the soma will result from, and be phase-locked to, oscillations in distal apical dendrite, making it possible for perisomatic spike generation to be gated by distal apical synaptic inputs (Richardson et al., 2003; Remme et al., 2009).

Alternatively, the soma could receive its own modulating signal that would effectively tune its frequency preference for input to distal apical regions. For example, a slow sinusoidal modulating signal could cause the soma's membrane potential to oscillate between hyperpolarized potentials and some level of depolarized potential depending on the amplitude of the modulation. At the peaks and troughs of this modulation, the soma would be most responsive to distal apical input within distinct frequency ranges, such as low alpha at the troughs and mid-high alpha at the peaks. Somatic frequency preference to distal apical input would therefore be phase-locked to the signal modulating somatic membrane potential. This type of process has implications for the ways in which multi-frequency coupling could occur in the brain (VanRullen and Koch, 2003). In short, these results indicate that the interactions between different functional zones should be considered in a more complete understanding of neuronal integration. Resonance analysis, in particular transfer resonance, may, therefore be a useful tool for assessing the integration of inputs across the entire neuronal membrane.

# Calcium Conductances [gCa(H) and gCa(L)] Amplify HCN and Muscarinic Resonance

The resonance observed in our model at hyperpolarized and resting membrane potentials is mediated by two currents: hyperpolarization-activated cyclic nucleotide-gated nonselective cation (HCN) and low-threshold calcium (Hutcheon et al., 1994, 1996; Hutcheon and Yarom, 2000; Ulrich, 2002). HCN (gh) generates resonance below about −60 mV, the strength of which is highly dependent on location due to the exponential gradient of increasing conductance density along the soma-apical dendrite axis (Narayanan and Johnston, 2008; Zhuchkova et al., 2013). Low-threshold calcium [gCa(L)] generates a resonance within a narrow band of membrane potentials (approximately −80 to −65 mV) where the activation and inactivation functions overlap (Figure S10 Supplementary Materials). The inactivation curve generates the resonance while the activation curve amplifies it. The strength of the resonance amplified by gCa(L) in our model is also highly dependent on location due to the distal region of high-density for the Ca2<sup>+</sup> ionic conductances.

Our results indicate that the high conductance density of gCa(L) present in the distal apical compartments in our model effectively amplifies subthreshold resonance, both in soma and distal apical regions, and increases the preferred input frequency of distal apical regions by up to 2 Hz in our model. The largest impact of gCa(L) occurs within the window current for this conductance, centered at −70 mV. This finding is consistent with the observation that gCa(L) and g<sup>h</sup> act together to produce the slow depolarization that underlies burst firing in some neocortical pyramidal neurons that are excited from hyperpolarized/resting membrane potentials (Foehring and Wyler, 1990; Foehring and Waters, 1991).

Resonance in our model at depolarized membrane potentials is mediated by the muscarinic (K+) current, or M current [gK(M)]. For example, in the somatic compartment of our model, this current produces a voltage-dependent resonant frequency that varies from approximately 8 Hz at resting membrane potentials to 23 Hz at −40 mV (see **Figure 3A**, black line), which agrees well with the resonant characteristics attributed to the M current in neocortical pyramidal neurons (Gutfreund et al., 1995; Hutcheon and Yarom, 2000). A well-known amplifier of the gK(M) resonance is the persistent sodium conductance [gNa(P)] (Hutcheon and Yarom, 2000). The reductions to input and transfer resonance quality, and input and transfer impedance at −20 mV observed in this study indicate that another important amplifier of gK(M) resonance is the high-threshold calcium conductance [gCa(H)]. This amplification peaks at −20 mV, the membrane potential at which gCa(H) is activating at the fastest rate (largest slope in activation function) and experiences its maximum time constant of 2.1 ms in our simulations (Figure S11 Supplementary Material).

The low membrane voltage ( < −60 mV) resonance mediated by g<sup>h</sup> and gCa(L) is strongest in distal apical regions where both conductances have high densities. On the other hand, the depolarized resonance ( > −60 mV) mediated by gK(M) and amplified by gNa(P) and gCa(H) is strongest in the perisomatic region of the neuron (compartments 0–2 in Table S1), because this is where gK(M) and gNa(P) conductance densities are highest. This type of complementary resonance in the 3–12 Hz range has been observed in (and modeled for) CA1 hippocampal pyramidal neurons (Hu et al., 2009). The gK(M) resonance in our study was observed to also extend into the range of beta frequencies (for example, see **Figure 3A**, black line), mostly due to the higher levels of membrane depolarization that we examine in this study. Extending the range of membrane potentials that our compartments were varied allowed us to identify the amplifying effect gCa(H) has on gK(M) resonance. It has been noted before that gCa(H) has the kinetics to qualify it as an amplifier of resonance (Hutcheon and Yarom, 2000). Our results suggest a critical role for the interaction of gCa(H) with gK(M) in the electrical coupling of distal apical and perisomatic regions of layer 5 pyramidal neurons—the amplification of gK(M) resonance by gCa(H).

# CONCLUSION

We have shown that tuning of a neuron's post-synaptic physiological properties to enhance association between distant inputs across the neuronal membrane impacts resonance. Our results indicate that interactions between different functional zones need to be considered in a more complete understanding of neuronal integration and that resonance analysis may be a useful tool for assessing the integration of inputs across the entire

### REFERENCES

Buzsaki, G. (2011). Rhythms of the Brain. New York, NY: Oxford University Press.

neuronal membrane. The distinct zones that we have identified through resonance analysis are consistent with functional zones described by previous research, and the resonant interaction that we have observed between some of these zones has revealed new insights about the function of Ca2<sup>+</sup> ionic conductances within layer 5 pyramidal neurons. By examining changes to resonance quality and resonant impedance when the distal Ca2<sup>+</sup> hot zone is toggled, we showed that both gCa(H) and gCa(L) amplify resonance that is generated by two complementary conductances: g<sup>h</sup> which becomes active below resting membrane potentials and is concentrated in distal apical regions, and gK(M) which becomes active above resting membrane potentials and is concentrated in perisomatic regions. Reductions to both gCa(H) and gCa(L) densities in distal apical regions reduces amplification of these resonances and consequently, reduces the electrical coupling of distal apical and perisomatic regions of the neuron that is necessary for it to function as a coincidence detector for input to both of these regions. Natural next steps for this research include determining how resonance properties impact suprathreshold neuronal and network behavior.

# AUTHOR CONTRIBUTIONS

All authors contributed equally to experimental design and theoretical considerations. AY established single-neuron model and implemented all simulated current injection capabilities. MF ran all simulations and developed resonance analysis post-processing algorithms. All authors contributed to data interpretation and manuscript preparation.

### ACKNOWLEDGMENTS

This research was funded by the U.S. Department of the Army and sponsored by the Army Research Laboratory. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncom. 2018.00029/full#supplementary-material


Bower, J. M., and Beeman, D. (2003). The Book of GENESIS: Exploring Realistic Neural Models with the GEneral NEural SImulation System. Available online at: http://www.genesis-sim.org/GENESIS

resonance at rest predict oscillation properties near threshold. J. Physiol. 560(Pt 1), 89–110. doi: 10.1113/jphysiol.2004.069930


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Felton, Yu, Boothe, Oie and Franaszczuk. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# PV Interneurons: Critical Regulators of E/I Balance for Prefrontal Cortex-Dependent Behavior and Psychiatric Disorders

Brielle R. Ferguson1,2\* and Wen-Jun Gao<sup>1</sup> \*

<sup>1</sup>Department of Neurobiology and Anatomy, College of Medicine, Drexel University, Philadelphia, PA, United States, <sup>2</sup>Department of Neurology and Neurological Sciences, School of Medicine, Stanford University, Stanford, CA, United States

Elucidating the prefrontal cortical microcircuit has been challenging, given its role in multiple complex behaviors, including working memory, cognitive flexibility, attention, social interaction and emotional regulation. Additionally, previous methodological limitations made it difficult to parse out the contribution of certain neuronal subpopulations in refining cortical representations. However, growing evidence supports a fundamental role of fast-spiking parvalbumin (PV) GABAergic interneurons in regulating pyramidal neuron activity to drive appropriate behavioral responses. Further, their function is heavily diminished in the prefrontal cortex (PFC) in numerous psychiatric diseases, including schizophrenia and autism. Previous research has demonstrated the importance of the optimal balance of excitation and inhibition (E/I) in cortical circuits in maintaining the efficiency of cortical information processing. Although we are still unraveling the mechanisms of information representation in the PFC, the E/I balance seems to be crucial, as pharmacological, chemogenetic and optogenetic approaches for disrupting E/I balance induce impairments in a range of PFC-dependent behaviors. In this review, we will explore two key hypotheses. First, PV interneurons are powerful regulators of E/I balance in the PFC, and help optimize the representation and processing of supramodal information in PFC. Second, diminishing the function of PV interneurons is sufficient to generate an elaborate symptom sequelae corresponding to those observed in a range of psychiatric diseases. Then, using this framework, we will speculate on whether this circuitry could represent a platform for the development of therapeutic interventions in disorders of PFC function.

Keywords: excitation/inhibition balance, GABA, PV interneurons, prefrontal cortex, cognition, psychiatric disorders

# INTRODUCTION

Cognitive impairment stifles independance by making even the simplest everyday tasks seem challenging and burdensome. Through limiting one's ability to focus, encode, retain and manipulate information to make menial decisions, simple endeavors like traveling from one place to another, or completing a typical work assignment become arduous. This decreases productivity, and forces afflicted individuals to rely partially or wholly on the care of others. For most, this loss of independance can be crippling to one's morale, resulting in increasing isolation

#### Edited by:

Vito Di Maio, Istituto di Scienze Applicate e Sistemi Intelligenti Eduardo Caianiello (CNR), Italy

#### Reviewed by:

Aleksey V. Zaitsev, Institute of Evolutionary Physiology and Biochemistry (RAS), Russia Lucy Maree Palmer, Universität Bern, Switzerland Scott S. Bolkan, Princeton University, United States

#### \*Correspondence:

Brielle R. Ferguson briellef@stanford.edu Wen-Jun Gao wg38@drexel.edu

Received: 08 February 2018 Accepted: 17 April 2018 Published: 16 May 2018

#### Citation:

Ferguson BR and Gao W-J (2018) PV Interneurons: Critical Regulators of E/I Balance for Prefrontal Cortex-Dependent Behavior and Psychiatric Disorders. Front. Neural Circuits 12:37. doi: 10.3389/fncir.2018.00037 from family, friends and society. Not surprisingly, disorders with reduced cognitive ability share high comorbidity with decreased sociability, as well as anxiety and depression. What remains unclear is whether these behaviors rely on a common circuitry, and if that circuitry could represent a platform for development of therapeutic interventions.

A large body of research suggests that the prefrontal cortex (PFC) is critical in regulating these behaviors and that the balance between excitatory and inhibitory neurotransmission (E/I balance) plays a fundamental role. The PFC is a unique associative cortical region that receives multimodal inputs from sensory regions, limbic structures and neuromodulatory nuclei (Carmichael and Price, 1996; Hoover and Vertes, 2007), and is taxed with processing these heterogeneous inputs to effectively guide ongoing behavior. The seamless execution of a range of behaviors relies on the integration of past experiences and current goals to select appropriate behavioral programs. Across species, we deem the myriad abilities linked to the PFC as ''executive functioning,'' and this category includes working memory, cognitive flexibility, planning, errormonitoring, decision-making, attention and social cognition. Although we have yet to unravel the precise mechanisms of information representation and processing in the PFC, recent optogenetic and imaging breakthroughs along with decades of lesion, pharmacological, and electrophysiological data highlight certain fundamental principles that may have relevance across PFC-dependent functions. Here, we argue that the E/I balance is essential for the proper execution of a range of PFC-dependent behaviors, and targeting this balance may be an effective approach in disorders that harbor related behavioral impairments.

We will begin with a discussion of GABAergic signaling and E/I balance using foundational research from sensory cortices to inform our interpretation of data from the PFC. Then, we will summarize data supporting our hypotheses, with a primary focus on research in rodents, as currently these types of studies allow for the most experimental flexibility and direct manipulation of specific neuronal types that regulate E/I balance. We acknowledge that the agranular rodent frontal cortex has no apparent structural equivalent of the dorsolateral PFC (dlPFC), a region in humans and primates linked to many of the behaviors that will be discussed in this review. Additionally, the human and non-human primate PFC are substantially more developed and complex relative to the comparable region in the rodent. Nevertheless, using a combination of tracing studies and lesion-symptom mapping, many scientists have concluded that the prelimbic region of the rodent medial PFC (mPFC) serves as the closest functional homolog to the dlPFC in humans and non-human primates (Uylings et al., 2003; Seamans et al., 2008). Human and primate studies will be utilized for comparison and framing purposes, but a detailed review of data from all three species is beyond the scope of this review. For a discussion involving a more comprehensive overview of clinical research discussing E/I balance and implications for psychiatric disease, see the following recent reviews (Foss-Feig et al., 2017; Krystal et al., 2017; Takarae and Sweeney, 2017).

# GABAergic SIGNALING AND E/I BALANCE

Within the cortex, there are two primary types of neurons: glutamatergic excitatory pyramidal neurons and GABAergic inhibitory interneurons. While a network of purely excitatory connections offers little computational complexity, GABAergic interneurons confer a circuit with tremendous flexibility, dynamically modulating the gain of pyramidal neuron responses from simple all-or-none responsiveness (Klausberger et al., 2003; Isaacson and Scanziani, 2011). Through inhibition of neighboring pyramidal neurons, GABAergic interneurons act as an important brake on excitatory signaling, control spike generation and timing (Pouille and Scanziani, 2001), and prevent incoming excitation from afferent structures from causing runaway feed-forward excitation (Dichter and Ayala, 1987).

GABAergic interneurons are a diverse family that can be classified by their morphology, electrophysiological properties, or histological markers (Markram et al., 2004, 2015). The most common nomenclature segregates interneurons into three broad types, parvalbumin (PV), somatostatin (SST) and ionotropic serotonin receptor 5HT3a (5HT3aR) expressing interneurons (Rudy et al., 2011). Although not entirely distinct, this population minimizes overlap between the groups and accounts for nearly the entirety of known interneuronal subtypes.

The most common subtype, PV interneurons, are known for their fast-spiking phenotype, low input resistance, and high-amplitude rapid after-hyperpolarization (AHP; Kawaguchi et al., 1987; Kawaguchi and Kubota, 1997). This combination of properties confers an ability to fire a rapid train of action potentials unlike any other neuron in the cortex. PV interneurons can be further divided into two subtypes: basket cells that innervate the soma and proximal dendrites, and chandelier cells that synapse onto the axon initial segment (Kawaguchi and Kubota, 1997; Petilla Interneuron Nomenclature Group et al., 2008). Relatively less is known about the properties and function of chandelier cells, and data demonstrate their action may not be purely inhibitory at the axon initial segment (Szabadics et al., 2006). PV basket cells are more common, and by virtue of being easier to identify and study, their properties and role in circuit integration in the normal brain have been characterized in greater detail. Data from paired recordings indicate that a single PV interneuron contacts nearly every local pyramidal neuron providing PV basket cells with an unparalleled ability to regulate the activity of nearby pyramidal neurons (Packer and Yuste, 2011). This allows for a tremendous level of feedforward and feedback inhibition that serves several important functions (Hu et al., 2014).

An ability of PV interneurons that appears to be universal across cortical regions is controlling spike timing in neighboring excitatory neurons (Pouille and Scanziani, 2001; Wehr and Zador, 2003). Pyramidal neurons receive constant barrages of excitatory synaptic input or excitatory post-synaptic potentials (EPSPs), and are tasked with deciding whether or not to fire an action potential. Given the density of excitatory connections both from afferent structures and within the local circuit, along with the high level of synaptic divergence, excitation of a sufficient magnitude to generate an action potential will likely recruit a large portion of the cortical network, making it challenging to represent information with any specificity. This creates two computational dilemmas, creating a network state such that all excitatory neurons are not firing constantly, as well as allowing for distinct neurons or groups of neurons to have differential responsiveness to EPSPs. One issue is addressed by feedforward inhibition (FFI), a principle illustrated canonically in thalamocortical circuits. A major source of excitatory cortical input is the thalamus, whose neurons branch and synapse onto both excitatory and inhibitory neurons. Seminal physiological studies demonstrated that when thalamic fibers are electrically stimulated, this causes a direct excitatory response in cortical pyramidal neurons, followed shortly by a strong hyperpolarization due to the activation of neighboring PV basket cells (Agmon and Connors, 1991; Castro-Alamancos and Connors, 1997). This enhances the temporal fidelity of pyramidal neuron responsiveness by limiting the window in which incoming excitation can generate an action potential, and has also been observed in the mPFC (Cruikshank et al., 2012; Delevich et al., 2015). The second issue is mitigated through another property of PV interneuron activity, gain control, or ''the rate at which the firing of a neuron increases in response to increasing excitatory input'' (Isaacson and Scanziani, 2011). This transforms pyramidal neuron responsiveness from a simple go or no-go pattern of activity to having the capacity to represent a broad range of input levels in their firing patterns, a feature that is likely to be crucial in the mPFC, which processes complex multimodal information.

By innervating pyramidal neurons at the soma, PV basket cells are strategically positioned to exert both FFI and gain control. Additionally, PV basket cells have been implicated in the generation of gamma oscillations (30–80 Hz, Buzsáki and Draguhn, 2004), an oscillation range linked to cognition and information processing across species (Howard et al., 2003; Gaetz et al., 2011; Lundqvist et al., 2016). PV basket cells have the ability to fire at frequencies corresponding to gamma oscillations (Gulyás et al., 2010). These interneurons also synapse primarily onto alpha1-gamma-Aminobutyric acid receptors (α1-GABAARs) that are present at high levels on the soma (Fritschy and Mohler, 1995; Nusser et al., 1996), and have a decay constant corresponding to the rising phase of gamma waves (Gonzalez-Burgos and Lewis, 2008). Demonstrating a definitive link, a pair of studies utilized a battery of optogenetic tools to silence or activate cortical PV interneurons, and found them necessary and sufficient in the generation and maintenance of gamma oscillations (Cardin et al., 2009; Sohal et al., 2009). Increasing their activity not only amplified cortical gamma oscillations, but also enhanced information processing by increasing the gain of incoming information and improving behaviors dependent on the manipulated brain regions. Data also indicate the presence of significant gamma alterations in diseases with cognitive impairment, such as schizophrenia and autism spectrum disorders (ASD; Rojas and Wilson, 2014; McNally and McCarley, 2016).

In order for these processes to occur efficiently, levels of inhibition and excitation must remain in the appropriate balance, which in the mPFC has been measured as approximately 20/80% (excitation/inhibition; Le Roux et al., 2008; den Boon et al., 2015). This balance is not static and requires that inhibition be responsive to fluctuations in cortical state and levels of excitatory input (Galarreta and Hestrin, 1998; Shu et al., 2003; Atallah and Scanziani, 2009; Xue et al., 2014). PV interneurons are well suited for helping maintain the proper balance, as their activity is associated with ''divisive'' rather than subtractive inhibition, which would allow PV interneurons to respond proportionally to dynamic excitation within cortical circuits (El-Boustani and Sur, 2014). Within this balance, however, there must be the capacity for change or plasticity. Data suggests that E/I balance helps facilitate the induction of long-term potentiation, which could be critical in allowing the PFC to remain flexible and adapt to new stimulus-response contingencies (Staff and Spruston, 2003; Marder and Buonomano, 2004).

## PHARMACOLOGICAL MANIPULATION OF E/I BALANCE AND BEHAVIOR

Two cognitive abilities that are among the most consistently associated with damage to the PFC are working memory (Kolb et al., 1974; Stokes and Best, 1990; Kesner et al., 1996) and cognitive flexibility (Ragozzino et al., 1999; Birrell and Brown, 2000; Block et al., 2007; Bissonette et al., 2008). Working memory is the ability to maintain, manipulate and recall information to guide behavior (Baddeley, 1992; Goldman-Rakic, 1994), while cognitive flexibility involves the capacity to update strategies in order to obtain a reward (Bissonette et al., 2013). Both represent key building blocks of higher-level cognitive processes and are often severely impacted in the context of psychiatric disease. Social cognition is similar in denotation to working memory, but applies the principles of acquiring, storing, and manipulating information specifically to the context of interacting in a flexible and appropriate manner with conspecifics (Adolphs, 1999). Supporting its reliance on the frontal cortex are lesion data from humans and non-human primates indicating that damage to the ventromedial and orbital frontal cortex consistently result in abnormal social behavior (Ackerly and Benton, 1948; Butter and Snyder, 1972).

PFC infusion of bicuculline, a GABAAR antagonist, disrupts working memory and cognitive flexibility in multiple species (Sawaguchi et al., 1988, 1989; Enomoto et al., 2011; Paine et al., 2011; Auger and Floresco, 2015). In vivo multi-unit recordings in primates revealed that following bicuculline administration (which would increase the E/I balance), putative FS interneurons that were not linked to any specific task epochs of a spatial working memory task begin to show spatial tuning. Additionally, pyramidal neurons that once exhibited directional sensitivity, began to respond to their non-preferred directions, and neurons that had no task-modulated activity respond erroneously to random directions and task epochs (Rao et al., 2000). In concert, these activity alterations would greatly reduce the signalto-noise ratio, severely limiting the ability of the circuit to generate and relay correct motor commands with precision. Circuits supporting normal patterns of social interaction may be regulated similarly, as bicuculline infusion into the rat mPFC reduces the number and duration of interactions as well as weakens the preference for a social stimulus compared to a non-social object (Cochran et al., 2002).

Conversely, studies that utilize enhancement of GABAergic transmission have been informative, as this approach would also disrupt the E/I balance. Muscimol, a GABA<sup>A</sup> agonist, is used commonly as a means for transient inactivation of brain regions. Direct infusion in the mPFC in rodents has resulted in impairments in working memory (Urban et al., 2014), retention of strategy-switching behavior (Rich and Shapiro, 2007) and social play behavior (van Kerkhof et al., 2013, 2014).

# OPTOGENETICS, IMAGING AND PHYSIOLOGY HIGHLIGHT THE ROLE OF GABAergic SIGNALING IN PFC-DEPENDENT BEHAVIOR

Advances in tools available for bidirectional modulation of neuronal activity with cell-type specificity have expanded our understanding of the role of PV interneurons in regulating PFC E/I balance and PFC-dependent behaviors. Additionally, the development of genetically encoded calcium indicators, like gCaMP (Chen et al., 2013), allows researchers to monitor activity or calcium transients selectively in PV interneurons or other neuronal subtypes. With these technologies, researchers can catalog the entire population of mPFC interneurons from learning and strategy acquisition through successful performance. Many findings confirm and fine-tune long-standing hypotheses about how PV interneurons regulate pyramidal neuron activity, oscillatory dynamics, and cortical information representation, but allow for more detailed investigation of their causal role in behavior with enhanced temporal and cellular control.

One extremely useful feature of optogenetics (Deisseroth, 2011) has been the in vivo identification of different neuronal subtypes, or ''optical tagging'' (Zhao et al., 2011; Roux et al., 2014), allowing researchers to confirm the circuit consequences of PV interneuron activation as well as how these interneurons contribute to particular behaviors using recording or imaging. For example, optical tagging revealed that PV interneurons provide rapid synchronous inhibition of mPFC excitatory neurons in contrast with the more variable and less forceful inhibition provided by SST interneurons (Kvitsiani et al., 2013). A similar approach was used to demonstrate that PV interneurons increase their firing during goal-directed behavior. In this same study, researchers also showed that increases in PV activity were correlated with inhibition of certain pyramidal neurons and enhancement of others, potentially representing ensembles of relevant vs. irrelevant information for successful responding respectively (Kim H. et al., 2016). Increases in PV activity have also been implicated in the extinction of cue-based responding (Sparta et al., 2014), a process that may mediate rule-shifts during cognitive flexibility tasks.

During working memory assays, PV interneurons in the mPFC fire prominently throughout short delays (5 s or less) between sample and choice presentation (Kim D. et al., 2016), and show preferential firing toward go trials in go, no-go task (Kamigaki and Dan, 2017). However, more complicated tasks reveal they may encode information about distinct task epochs. Lagler et al. (2016) utilized a multidimensional delay task, where a differential gustatory cue signaled the location of reward in a Y-maze, and quantified PV interneuron activity during discrete phases of the assay. Interestingly, mPFC PV interneurons exhibited episodic modulation, and a large group showed preferential activation during the ''goal run, '' when the mouse was proceeding through the base of the Y-maze before making a choice (Lagler et al., 2016). PV interneurons in the mPFC also increase their firing in response to typical social interactions, compared to the investigation of a non-social object (Selimbeyoglu et al., 2017).

Modulation of PV activity has also been useful in demonstrating one of the more obvious principles of E/I balance, that too much inhibition can also impair PFC circuit function. Optogenetic or pharmacogenetic activation of PV interneurons has been an effective method of disrupting the function of particular brain regions, suggesting that alterations in E/I balance in either direction can alter behavior. We recently demonstrated that increasing the excitability of mPFC PV interneurons (thereby reducing the E/I balance) significantly impairs working memory, cognitive flexibility and social interaction (Ferguson and Gao, 2018). Similarly, optogenetic activation of PV interneurons or GABAergic neurons more broadly disrupts delay-related activity and working memory performance (Rossi et al., 2012; Liu et al., 2014). These data corroborate pharmacological studies using muscimol, a GABA agonist, as a method of inactivating PFC to elucidate its role in various behaviors (Narayanan and Laubach, 2006; Urban et al., 2014). Correspondingly, working memory and social interaction deficits can be recapitulated by enhancing pyramidal neuron excitability (enhancing E/I balance; Yizhar et al., 2011; Liu et al., 2014; Kamigaki and Dan, 2017).

# BUILDING THE mPFC MICROCIRCUIT

In combination with numerous elegant studies from the visual, auditory and somatosensory systems, we can begin to develop a picture of how the mPFC microcircuit might be organized to represent and process information relevant for executive function. The largest source of afferents originates from the mediodorsal thalamus (MD), but mPFC neurons also receive convergent inputs from limbic structures including the hippocampus and amygdala, that may convey spatial and emotional information. Additionally, the mPFC has dense innervation from neuromodulatory regions, the ventral tegmental area (VTA), locus coeruleus (LC), dorsal raphe (DR) and basal forebrain (BF), which may influence the state of the animal including level of attention, arousal, or the salience of the current task or resulting rewards or punishment (Kuroda et al., 1996; Hoover and Vertes, 2007). Using working memory as an example, early studies in rodents found that neurons representing differential task-related information in a consistent manner were sparse (Jung et al., 1998, 2000; Baeg et al., 2003).

However, more recent work has identified numerous mPFC neurons that change their firing frequency during distinct phases of a delayed-alternation task. Of these neurons, Yang et al. (2014) was able to separate these into units showing preferential firing on left vs. right trials, and encode information such as the choice, delay, and presence of reward. Other studies highlight a segregation of spatial and rule information encoding in mPFC neurons, of which the rule representation depends critically on the activity of the MD (Bolkan et al., 2017; Schmitt et al., 2017). Taken together, this suggests that pyramidal neurons in the mPFC form functional units that represent different features relevant to successful working memory performance and other PFC-dependent behaviors (**Figure 1**).

How might the appropriate E/I balance be maintained in the mPFC? As a general principle, thalamic inputs provide stronger activation to PV interneurons vs. pyramidal cells in sensory cortices (Hestrin, 1993; Kloc and Maffei, 2014). Similarly, enhancing excitability of MD neurons results in enhanced spiking of PV interneurons in mPFC, and not pyramidal cells (Schmitt et al., 2017), while dampening MD activity increases E/I balance by selectively reducing inhibitory currents onto pyramidal cells (Ferguson and Gao, 2018). Thus, the MD seems to function drive PV interneurons in the mPFC to maintain an E/I balance biased towards inhibition. Maintaining a network state with a high level of inhibition, would strengthen computational capacity through enhancing temporal precision and increasing the gain of pyramidal neuron responsiveness, as well as enhancing the signal to noise ratio of neurons encoding task or rule representations (**Figure 1**). Supporting that this large-scale inhibition is mainly PV interneuron mediated, we observed that enhancing PV interneuron excitability can fully normalize shifts in E/I balance with corresponding behavioral improvements (Ferguson and Gao, 2018).

PV interneurons are powerful regulators of pyramidal neuron activity and appear to be the most vulnerable across psychiatric disorders, but research highlights other major inhibitory interneurons that help fine-tune circuits supporting PFC-dependent behaviors (Wang et al., 2004). Axo-somatic inhibition accounts for only up to 10% of total inhibition received by excitatory neurons, while the remaining 90% occurs at proximal and distal dendrites and spines and is mediated by other inhibitory neuronal subtypes (Kubota et al., 2016). For example, SST interneurons exhibit phasic firing during distinct phases of a delayed alternation task, and seem to show segregation of firing among left vs. right trials (Kim D. et al., 2016). Further, in the hippocampus, SST interneurons provide powerful dendritic inhibition that helps to regulate both synaptic integration as well as the gain of pyramidal neuron responsiveness (Lovett-Barron et al., 2012). Additionally, vasoactive-intestinal peptide (VIP, 5HT3aR expressing) interneurons (Férézou et al., 2002), which inhibit both SST and PV interneurons seem to serve an important disinhibitory function in cortical circuits (Pi et al., 2013), that may help modulate the gain of pyramidal neuron responses (Fu et al., 2014). Recent data demonstrates VIP interneuron activity enhances mPFC circuit performance during two delay tasks involving working memory (Kamigaki and Dan, 2017). VIP interneurons may also facilitate forming associations in pyramidal neurons, as VIP interneurons show the greatest increase in activity along with pyramidal neurons in a go/nogo task following punishment (Pinto and Dan, 2015). As they inhibit both PV and SST cells, disinhibition of pyramidal cells by VIP interneurons may be a mechanism for facilitating the activity of neuronal ensembles relevant for optimal task performance in multiple PFC-dependent tasks (**Figure 1**). Further, targeting VIP interneuron activity has shown promising results in an animal model of hypofrontality in schizophrenia (Koukouli et al., 2017).

#### TARGETING E/I BALANCE IN ANIMAL MODELS TO IMPROVE BEHAVIOR

Multiple approaches for enhancing the normal function of prefrontal PV interneurons have been effective in mitigating a constellation of behavioral symptoms animal models of cognitive dysfunction and social abnormalities. For example, altering expression of Dlx5/6, a protein that regulates PV interneuron development leads to cognitive and other behavioral abnormalities in mice. Optogenetic activation of mPFC PV interneurons in Dlx5/6 mice is sufficient to ameliorate cognitive flexibility and social interaction deficits (Cho et al., 2015). Similarly, mice deficient in Contactin-associated protein-like (CNTNAP)–2 display marked social impairments that can be rescued by increasing PV interneuron activity optogenetically (Selimbeyoglu et al., 2017) in agreement with findings from naïve mice with optogenetically disrupted mPFC E/I balance (Yizhar et al., 2011). Dampening activity in the MD (modeling findings of mediodorsal hypofunction in schizophrenia) has been shown to impair working memory, cognitive flexibility, social interaction and alter anxiety-related behavior (Parnaudeau et al., 2013, 2015; Bolkan et al., 2017; Ferguson and Gao, 2018). However, pharmacogenetic activation of mPFC PV interneurons normalizes all mPFC-dependent behavioral deficits (Ferguson and Gao, 2018), highlighting the therapeutic potential of PV interneurons as a strategy for repairing E/I balance and treating a range of behavioral deficits. These results are complemented by pharmacological augmentation of GABAergic signaling, most commonly with benzodiazepines, that has been shown to repair behaviors including social interaction and cognitive flexibility (Wen et al., 2010; Han et al., 2012; Cho et al., 2015).

An ongoing question has been whether altering the properties of PV interneurons is responsible for the initial manifestation of behavioral deficits in psychiatric disease and animal models, in particular, those beyond the cognitive realm. For example altering the excitability of PV interneurons by genetic deletion of N-methyl-D-aspartate receptors (NMDARs), leads to selective deficits in working memory and associative learning (Carlen et al., 2012), while a broader deletion in various subtypes of cortical and hippocampal interneurons, induces cognitive impairments along with anxiety-related behavior, depressive symptoms and social impairments (Belforte et al., 2010). Similarly, reducing the expression of the voltage-gated sodium channel (NaV) 1.1, a channel linked to autism-related behaviors in Dravet syndrome, broadly in forebrain GABAergic interneurons induces spatial memory and social interaction deficits, along with increased repetitive behaviors and anxiety (Han et al., 2012). Optogenetic suppression of PV interneuron activity recapitulates only cognitive impairments, specifically a deficit in extradimensional set-shifting (Cho et al., 2015). Additionally, selective expression of tetanus toxin light chain (TeLC) in mPFC PV interneurons, disrupts working memory and cognitive flexibility, while sparing behaviors representing positive and negative symptoms in schizophrenia (Murray et al., 2015). However, in this study, recording of mPFC local field potentials revealed gamma oscillations did not differ between groups, so without further physiological analysis, the ultimate circuit consequences are somewhat ambiguous. Although it is unclear whether disrupting PV activity can disrupt a broader range of behaviors given different findings with articles employing disparate methodologies, the specificity of the insult seems to be critical. Solely targeting PV interneurons will reliably recapitulate phenotypes of cognitive disruption, while broader interneuronal insults increase the likelihood of impacting other behaviors. Another possibility is that the magnitude of the PV interneuron deficit may correlate directly with the propensity for disrupting behaviors beyond cognition.

### E/I BALANCE IN PSYCHIATRIC DISEASE

The pattern of rodent PFC lesions and deficits mirrors the impairments observed following damage to the homologous structures in primates and humans, suggesting the functions of these brain regions are highly conserved across species. Even more compelling is how myriad findings recapitulate commonly observed morphological alterations and behavioral endophenotypes seen in psychiatric disease. One of the most demonstrative parallels is found in the pathophysiology and symptom sequelae of schizophrenia. Schizophrenia is a debilitating psychiatric disease that affects approximately 1.1% of the world's population, and is characterized by positive, negative, and cognitive symptoms (Regier et al., 1993). Working memory and cognitive flexibility represent core dysfunctions in schizophrenia that remain intractable by treatment with current antipsychotics (Lee and Park, 2005; Insel, 2010). These along with a range of additional cognitive symptoms emerge prior to the onset of psychosis in early adulthood, and are the largest predictor of functional outcome in individual patients (Green, 1996). Negative symptoms involve anhedonia and social withdrawal, and are also minimally responsive to presently available treatments (Remington et al., 2016).

Analogous to experimental observations from animal studies, parallel deficits in PFC GABAergic signaling (Lewis et al., 2012), and E/I balance (Lisman, 2012) are both implicated in the pathophysiology of cognitive dysfunction in schizophrenia. Functional imaging reveals that schizophrenics exhibit patterns of hypofunction in the PFC (Van Snellenberg et al., 2016) as well as reduced volume in frontal cortex postmortem tissue (Selemon et al., 2002). Copious research demonstrates reductions in the marker for the GABA synthesizing enzyme, GAD-67, in PV interneurons in the PFC (Akbarian and Huang, 2006; Lewis et al., 2012). Levels of GAD-67 are activity-dependent (Benson et al., 1989; Sloviter et al., 1996), and coupled with the common finding that PV levels are also decreased within these interneurons (Glausier and Lewis, 2017), it suggests that the function of PFC PV interneurons is diminished in schizophrenia. Correspondingly, patients with schizophrenia show decreases in task-evoked in gamma oscillations, an oscillation band believed to be dependent on PV interneuronal firing, that correlate with the level of functional impairment in working memory (Basar-Eroglu et al., 2007).

Analogous to human pathology, rodent models of cognitive dysfunction in schizophrenia show an extremely high prevalence of reductions in PV or GAD67 expression in the mPFC. As more evidence reveals this disorder likely stems from heterogeneous etiologies, pharmacological, environmental, and transgenic models exhibit this common feature (Cochran et al., 2002; Francois et al., 2009; Carlson et al., 2011), suggesting that GABAergic hypofunction represents a convergence point. The constellation of data from human populations, animal models, experimental disruption of GABAergic signaling, and electrophysiological recordings across these contexts, highlight this system as a pathway to intervention in afflicted individuals.

Intriguingly, this pattern repeats among multiple psychiatric disorders including ASD, depression, and intellectual disability (Gao and Penzes, 2015; Luscher and Fuchs, 2015). ASD in particular harbors significant overlapping behavioral impairments and underlying neurobiological alterations. Autistic patients also exhibit prominent deficits in executive function, including working memory impairments and behavioral inflexibility (Hughes et al., 1994). Social abnormalities are also a behavioral hallmark of autistic pathology, primarily manifested as a deficit in non-verbal communication (Mundy et al., 1986). Autistic patients also suffer from a greater level of anxiety and depression relative to general population (Kim et al., 2000).

E/I balance disruption has also emerged as a prominent hypothesis in ASD (Rubenstein and Merzenich, 2003). Dysregulation of GABAergic signaling has been implicated in the etiology of ASD, and disorders sharing high comorbidity with the disease including anxiety and epilepsy (Coghlan et al., 2012). The chromosomal region 15q11-q13 is comprised of multiple genes encoding subunits of the GABAAR. Microduplications in this region (Cook et al., 1998; Menold et al., 2001) or aberrant expression of the associated gene products (Hogart et al., 2007; Mendez et al., 2013) have been frequently observed in ASD clinical populations. In addition to the finding of copy number variations in the 15qllq13 chromosomal locus, the majority of autism-linked genes are preferentially expressed in interneurons (Xu et al., 2014).

Supporting this are in vivo findings of reduced GABA and GABAAR levels in the frontal cortex of autistic patients (Harada et al., 2011; Mori et al., 2012), along with reductions in gamma oscillations (Sun et al., 2012). Similar deficiencies in inhibitory neurotransmission have been reported in mice with mutations in ASD-linked genes, mice that also exhibit relevant behavioral impairments (Peñagarikano et al., 2011; Han et al., 2012). For example, T(+)Itpr3(tf)/J (BTBR) mice, a model of idiopathic autism, exhibit decreased GABAergic currents and increased excitatory neurotransmission in the hippocampus, indicating a shift in the E/I balance. Concurrent with these physiological changes, these mice display reductions in sociability, cognitive impairments, and alterations in anxiety-related behaviors (Han et al., 2014). Additionally, mouse models with disruption of genes such as methyl-CpG-binding protein-2 (MECP2), Scn1a+/<sup>−</sup> and CNTNAP–2, all harbor interneuron deficits along with behavioral impairments (Chao et al., 2010; Peñagarikano et al., 2011; Han et al., 2012). Modeling of Fragile X syndrome, a genetic disorder associated with intellectual disability, also reveals cognitive impairments are associated with prominent GABAergic hypofunction in mice (Selby et al., 2007; Curia et al., 2009).

# GABAergic SIGNALING AS A THERAPEUTIC TARGET

Multiple studies suggest that augmenting GABAergic signaling via PV interneuron modulation can be effective in ameliorating deficits in working memory, cognitive flexibility and sociability in animal models of psychiatric disease. This has substantial implications for schizophrenia, given that cognitive symptoms are treatment-resistant with both typical and atypical antipsychotics, and are a significant predictor of quality of life in individual patients (Green, 2006). These results complement a rich history of data and hypotheses surrounding GABAergic hypofunction in the human PFC representing a final common pathway in the cognitive symptoms in schizophrenia (Lewis, 2014). Yet, we still await the discovery of a GABAergic modulator that displays effectiveness in alleviating cognitive dysfunction across multiple disorders.

Of the recent clinical trials targeting cognitive symptoms, very few utilize GABAergic drugs, relative to agonists and modulators of glutamatergic signaling. Although benzodiazepines have shown potential in treating psychosis (Wolkowitz and Pickar, 1991; Carpenter et al., 1999), their potential in reducing cognitive impairments remains largely unexplored with the exception of a few studies. Interestingly in one study, a benzodiazepine, lorazepam, exacerbated working memory impairments in schizophrenic patients and healthy controls and altered activity within networks supporting cognition. Researchers concluded that hyper-inhibition was responsible for cognitive dysfunction, and correspondingly, flumazenil an antagonist of the benzodiazepine site attenuated deficits (Menzies et al., 2007). However, a separate study found iomazenil, a flumazenil analog, increased psychoses in schizophrenic patients (Ahn et al., 2011). Given the high comorbidity of ASD with epilepsy (Canitano, 2007), researchers have gleaned that drugs effective in reducing seizure activity that typically works through elevating GABA levels also mitigate autistic symptoms (Jambaqué et al., 2000). Promisingly, GABA agonists have reached Phase II clinical trials for the treatment of social disability in ASD, but so far have only focused on α2/α3-GABAARs.

The synapse between PV-expressing chandelier cells and the axon initial segment have become frequent but unsuccessful target for therapeutics in schizophrenia. In patients with schizophrenia, chandelier cell axon terminals exhibit decreased levels of the reuptake enzyme, GABA transporter 1 (GAT1) mRNA, along with higher expression of α2-GABAARs on the axon initial segment of pyramidal neurons (Volk et al., 2002). These alterations would result in less GABA reuptake, and increased post-synaptic GABAergic inhibition respectively, which together would serve to augment GABAergic neurotransmission at these particular synapses. This has been interpreted as a compensatory mechanism, implying these synapses may be the site of the initial GABAergic deficit (Volk and Lewis, 2005). However, despite promising results from early studies (Lewis et al., 2008), larger sample sizes have yielded no differences between schizophrenic patients treated with an α2/α3-GABAAR agonists compared to those treated with the placebo (Buchanan et al., 2011). Given the failure of enhancing α2-GABAAR-mediated signaling in improving cognitive dysfunction in clinical trials as well as in the context of compensatory mechanisms in schizophrenia, it is reasonable to conclude these receptors are not a viable therapeutic target.

One difficulty in utilizing GABAergic therapeutics, including benzodiazepines, is their sedative properties, which are mediated by their action at the α1 GABAAR subunit (Löw et al., 2000). This may account for the lack of exploration of α1-GABAAR modulators in clinical trials. However the actions of dopamine across different subjects and contexts provide a compelling example of how optimal levels of neurotransmitter can be associated with normal cognition, attention and alertness, but levels either too high or low, can result in impaired cognition, inability to focus, and drowsiness (Cools and D'Esposito, 2011). Our findings using indiplon, an α1-GABAAR positive allosteric modulator (Ferguson and Gao, 2018), and other studies using benzodiazepines (Han et al., 2012, 2014) suggest that the therapeutic window for influencing cognitive function may be distinct from doses that induce sedation. It is likely that GABAergic signaling in the PFC and its correlated functions follow an inverted-U trajectory, and with better biomarkers for individual differences in GABA levels, indiplon, novel α1- GABAAR modulators or benzodiazepines may harbor significant therapeutic potential.

# FUTURE DIRECTIONS

Additional questions remain that could help inform a circuit model of information processing in the mPFC. For example, how organized are the actions of PV interneurons in the mPFC? Are they providing a general blanket inhibition that helps maintain a cortical state conducive to information processing due to their properties described above? Evidence supporting this is imaging of PV interneurons revealing that PV interneurons show the largest modulation, and have uniform firing patterns that persist throughout delay periods during PFC-dependent tasks in comparison to other interneuron subtypes (Pinto and Dan, 2015; Kim D. et al., 2016). If PV interneurons do provide indiscriminate inhibition throughout the mPFC, to what extent to disinhibitory circuits through VIP and SST interneurons help further enhance information representation, and through what mechanisms and inputs? Other data indicate segregated groups of PV interneurons encode distinct task phases (Lagler et al., 2016), suggesting instead of blanket inhibition, patterns of activity are differentially regulated across different PV interneurons. Future studies should also further explore the actions of PV chandelier cells in the mPFC during behavior. New research indicates this subtype may preferentially inhibit amygdala-projecting pyramidal neurons in the mPFC, suggesting chandelier cells have a distinct role in the mPFC microcircuit from PV basket cells (Lu et al., 2017).

Currently available imaging and physiological methods provide promising approaches for measuring changes in neuronal activity that are correlated with successful behavior. However, given the diversity of neocortical interneuron subtypes and function, how PV interneurons and other cell types contribute to modulation of E/I balance, and whether and how modulation of each subtype impact different PFC-dependent behaviors warrants further investigation. We still lack effective treatments for cognitive deficits, social interaction impairments, and other associated behavioral dysfunctions in numerous psychiatric disorders, underscoring the importance of continuing to unravel how these behavioral processes occur under normal conditions. If researchers can identify how to effectively manipulate PFC circuit activity, we can develop reliable strategies for engineering optimal patterns of cortical activity to ameliorate performance. Ultimately, if successful manipulations can be linked to physiological signatures that can be observed using less invasive recording approaches such as oscillatory patterns, we can potentially determine biomarkers for successful behavioral therapies in humans.

### AUTHOR CONTRIBUTIONS

BF wrote the manuscript and W-JG edited it.

#### REFERENCES


## FUNDING

This work was supported by National Institute of Health (NIH)/National Institute of Mental Health (NIMH), F31MH 111361 to BF and R01MH085666 and R21MH111609 to W-JG and Helen S. Vernik Schizophrenia Pilot Research Project from the Department of Psychiatry, Drexel University College of Medicine to BF, Linda Chamberlin and W-JG (2017).

the treatment of cognitive impairments in people with schizophrenia. Biol. Psychiatry 69, 442–449. doi: 10.1016/j.biopsych.2010.09.052


schizophrenia. Biol. Psychiatry 80, 617–626. doi: 10.1016/j.biopsych.2016. 02.017


for neurogenetic disorders. J. Neurosci. 34, 1420–1431. doi: 10.1523/jneurosci. 4488-13.2014


**Conflict of Interest Statement**: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2018 Ferguson and Gao. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# How Memory Conforms to Brain Development

#### Ana P. Millán, Joaquín J. Torres\* and Joaquín Marro

Institute "Carlos I" for Theoretical and Computational Physics, University of Granada, Granada, Spain

Nature exhibits countless examples of adaptive networks, whose topology evolves constantly coupled with the activity due to its function. The brain is an illustrative example of a system in which a dynamic complex network develops by the generation and pruning of synaptic contacts between neurons while memories are acquired and consolidated. Here, we consider a recently proposed brain developing model to study how mechanisms responsible for the evolution of brain structure affect and are affected by memory storage processes. Following recent experimental observations, we assume that the basic rules for adding and removing synapses depend on local synaptic currents at the respective neurons in addition to global mechanisms depending on the mean connectivity. In this way a feedback loop between "form" and "function" spontaneously emerges that influences the ability of the system to optimally store and retrieve sensory information in patterns of brain activity or memories. In particular, we report here that, as a consequence of such a feedback-loop, oscillations in the activity of the system among the memorized patterns can occur, depending on parameters, reminding mind dynamical processes. Such oscillations have their origin in the destabilization of memory attractors due to the pruning dynamics, which induces a kind of structural disorder or noise in the system at a long-term scale. This constantly modifies the synaptic disorder induced by the interference among the many patterns of activity memorized in the system. Such new intriguing oscillatory behavior is to be associated only to long-term synaptic mechanisms during the network evolution dynamics, and it does not depend on short-term synaptic processes, as assumed in other studies, that are not present in our model.

#### Edited by:

Vito Di Maio, Institute of Applied Sciences and Intelligent Systems (ISASI), Italy

#### Reviewed by:

Andreas Knoblauch, Hochschule Albstadt-Sigmaringen, Germany Christian Tetzlaff, Max-Planck-Institute for Dynamics and Self-Organisation, Max Planck Society (MPG), Germany

#### \*Correspondence:

Joaquín J. Torres jtorres@onsager.ugr.es

Received: 13 September 2018 Accepted: 26 March 2019 Published: 16 April 2019

#### Citation:

Millán AP, Torres JJ and Marro J (2019) How Memory Conforms to Brain Development. Front. Comput. Neurosci. 13:22. doi: 10.3389/fncom.2019.00022

Keywords: brain developing, brain structure and function, synaptic pruning, storage capacity, dynamic memories

# 1. INTRODUCTION

A complex interrelation between "form" and "function" is known to play an important role in nature (Gross and Blasius, 2008; Vazquez et al., 2008; Sayama et al., 2013). The idea has been efficiently developed in the field of adaptive networks, in which a sort of coupling feedback loop sets in between the network dynamic activity and its topological structure. Outstanding phenomena then emerge, including self-organization into complex topologies that exhibit robust dynamics, spontaneous differentiation of the nodes, or complex mutual dynamics in both activity and topology, in any case mimicking many different conditions in nature (Bullmore and Sporns, 2009; Sayama et al., 2013; Millán et al., 2018a). This framework has revealed quite useful to understand fundamental questions concerning mammal brains, e.g., how structural and functional properties relate to each other both at the level of models involving sets of neurons and synapses and at the coarse-grained scale of connectomes and functional nets which is captured by imaging techniques (Bullmore and Sporns, 2009). A main question that we can thus address is how an efficient brain develops by synaptic pruning after a sort of "wild" proliferation of synaptic connections between neurons following conception (Chechik et al., 1998; Iglesias et al., 2005; Santos and Noggle, 2011; Presumey et al., 2017). In humans, for example, synaptic density at birth is about twice that at puberty, and certain brain disorders, such as autism spectrum disorder (ASD) and schizophrenia, have been related to details of synaptic pruning (Keshavan et al., 1994; Geschwind and Levitt, 2007; Faludi and Mirnics, 2011; Kolb et al., 2012; Fornito et al., 2015). In particular, ASD has been associated with a defect of synaptic pruning in certain brain areas (Tang et al., 2014), whereas schizophrenia could be related to an excessive pruning (Sekar et al., 2016). In any case, it now seems clear that such synaptic pruning involves in some way an optimization process, probably aimed at minimizing both energy consumption and the genetic information that otherwise would be needed to build an efficient and robust network (Chechik et al., 1999; Chklovskii et al., 2004; Johnson et al., 2010; Knoblauch et al., 2010; Navlakha et al., 2015). In particular, recent studies on associative memory have shown that this process could greatly improve memory retrieval under a noisy environment, such as it is the case in biological systems (Millán et al., 2018a). Moreover, ongoing structural plasticity in the adult brain has also been suggested to improve substantially the storage capacity (Chklovskii et al., 2004; Knoblauch et al., 2010), and has been related to graded amnesia, catastrophic forgetting, and the spacing effect (Knoblauch et al., 2014; Knoblauch and Sommer, 2016). These results are based on the fact that the number of potential synapses a neuron could develop, i.e., its potential connectivity, is much greater than the actual number of synapses, and structural plasticity allows the system to explore different wiring possibilities (Stepanyants et al., 2002; Fares and Stepanyants, 2009).

Here, we use an adaptive—sometimes also called co-evolving brain network model, which has already been used by us to describe synaptic pruning in humans (Millán et al., 2018a,b), to analyze how the dynamical processes of adding and removing synapses during brain development can affect the ability of the network to store and optimally retrieve a given set of memories. Our system combines the auto-associative Amari-Hopfield neural network (Amari, 1972; Hopfield, 1982) with a preferential-attachment dynamics for the network evolution in a way that has been shown to accurately reproduce the observed variation of neuron connectivity data on human brains during infancy (Johnson et al., 2010; Millán et al., 2018b). As empirically observed—see (Holtmaat and Svoboda, 2009) and references therein—this model assumes that the probabilities of growth and death of synapses depend on both the mean connectivity in the system and the neural activity. Previous studies have analyzed the effect of thermal noise in the system and its emergent behavior, and they have shown that the coupling between neuronal activity and connectivity creates a feedback loop between form and function since the system activity influences its topology and, in turn, it is affected by the network structure through the synaptic currents the neurons receive (Millán et al., 2018a). As a matter of fact, depending on parameters, this system is then able to produce heterogeneous networks with the presence of hubs, similar to the ones observed in actual neural systems (Van Den Heuvel and Sporns, 2011; Crossley et al., 2014; Oh et al., 2014; Stafford et al., 2014), with high memory retrieval and noise tolerance. Another recent work has also studied the effect of a transient period of high connectivity before synaptic pruning begins (Millán et al., 2018b), as observed in mammal brains (Huttenlocher and Dabholkar, 1997; Navlakha et al., 2015), demonstrating that it has beneficial effects for memory recovery and the emergence of an organized stationary state in the system.

Here we develop on the effect that synaptic (or quenched) disorder resulting from the interference among many patterns of activity—stored by Hebbian learning on the synaptic weights has on the emergent behavior of the system. We show that, as a consequence of the interplay between structural (i.e., pruning), thermal and quenched disorder, oscillations can emerge in the activity of the model which imply visiting different memorized patterns, an emergent behavior that had not been reported before in this model. This intriguing behavior is precisely due to long-term synaptic mechanisms associated with the network evolution dynamics, and not to short-term synaptic processes, such as synaptic depression and facilitation (Pantic et al., 2002; Marro et al., 2007; Torres et al., 2007, 2008; Torres and Marro, 2015) or spike frequency adaptation (Knoblauch and Palm, 2002; Ha and Cheong, 2017), which are not present in our model. These have already been described to induce oscillations among stored patterns of network activity, however the biophysical mechanisms behind them are different from the topological rewiring process considered here, and in particular they act on shorter time-scales—on the order of ms as opposed to the time scale of hours or days in which synaptic rewiring operates. It would be straightforward to extend the present study by adding short-term mechanisms, and we hypothesize that the interplay between different neuron and synaptic processes during learning and brain evolution could give rise to other types of oscillatory phenomena associated with non-equilibrium phases not yet reported, a fact that we glimpse could have strong computational implications.

#### 2. MODEL AND METHODS

Our system consists in a time-dependent, symmetric, undirected, N-node complex network (Boccaletti et al., 2006) of neurons, defined at time t by the adjacency matrix e(t), with elements eij(t) = {0, 1}, in which each node represents a neuron and each edge [eij(t) = 1] stands for a synapse. The degree of node i at time t is defined as

$$k\_i(t) = \sum\_{j=1}^{N} e\_{ij}(t) \tag{1}$$

and the mean degree of the network is

$$\kappa(t) = \frac{1}{N} \sum\_{i=1}^{N} k\_i(t). \tag{2}$$

Following a familiar (Amari-Hopfield) prescription (Amari, 1972; Amit, 1989), each neuron i is modeled as a stochastic binary unit, si(t) = {0, 1} (representing respectively a silent and a firing neuron), whose state evolves in time according to the probabilistic dynamics

$$P\left[s\_i(t+1) = 1\right] = \frac{1}{2}\left\{1 + \tanh\left[T^{-1}\left(h\_i(t) - \theta\_i(t)\right)\right]\right\},\tag{3}$$

where

$$h\_i(t) = \sum\_{j=1}^{N} \omega\_{ij} e\_{ij}(t) s\_j(t) \tag{4}$$

is the local field at neuron i quantifying the incoming input from neighbor neurons and

$$\theta\_i(t) = \frac{1}{2} \sum\_{j=1}^{N} w\_{ij} e\_{i\vec{j}}(t) \tag{5}$$

is the neuron'sthreshold for firing. This definition of the threshold is typically considered, in the case of static networks, when the more biologically plausible {0, 1} code is used instead of the canonical {±1} one, since it allows one to recover the phase diagram of the canonical, fully connected Amari-Hopfield model (Amit, 1989). Therefore, we maintain it when extending the model to a time dependent topology, and it naturally leads to a dynamic threshold. This is not a strong assumption since dynamic or adaptive thresholds have been widely described in several neural systems. For instance, they have been shown to create a nontrivial motion between the attractors of the system (Horn and Usher, 1989; Itskov et al., 2011) and to have a major role in stochastic resonance (Mejias and Torres, 2011) and in the functioning of sensory systems (Fricker et al., 1999; Azouz and Gray, 2000, 2003; Cardin et al., 2008; Kobayashi et al., 2009). Mechanisms of threshold adaptation have been found to help to avoid saturating activity during developmental changes (Turrigiano et al., 1998), and to be related to homeostatic regulation mechanisms observed in cortical neurons (Abbott and LeMasson, 1993; Turrigiano et al., 1998), and to the emergence of self-organized criticality in neural systems (Uhlig et al., 2013; Hobbiss et al., 2018). In our context, θi(t) depends only on the existing synapses, which can be seen as a means of homeostasis since the response of a neuron is regulated by the number and strength of its synaptic contacts, thus avoiding silencing lowdegree neurons and saturation of hubs. Furthermore, in our model the term eijwij in Equation (5) characterizes the intensity of the synaptic transmission between neurons i and j, so that the threshold dynamics depends indirectly on the neural activity.

On the other hand, the noise parameter ortemperature T (T > 0) sets the level of stochasticity on the activity of the neurons, so that if T = 0 the evolution of the system is deterministic and the state of a neuron at time t is completely determined by the states of its neighbors at time t−1. For T > 0, however, the evolution is stochastic and, as T is increased, the thermal noise has a stronger effect. The strength of each synapse, or its synaptic weight, wij, is a real variable defined by means of a set of P binary patterns of neural activity, ξ µ <sup>i</sup> ∈ {0, 1}, µ = 1, ..., P, according to the Hebbian learning prescription (Amit, 1989),

$$\begin{aligned} \boldsymbol{w}\_{i\bar{j}} &= [\boldsymbol{\kappa}\_0 \boldsymbol{a}\_0 \,(1 - \boldsymbol{a}\_0)]^{-1} \sum\_{\mu=1}^p \left(\boldsymbol{\xi}\_i^{\mu} - \boldsymbol{a}\_0\right) \left(\boldsymbol{\xi}\_j^{\mu} - \boldsymbol{a}\_0\right), \; i \neq j \\ \boldsymbol{w}\_{i\bar{i}} &= \boldsymbol{0}, \end{aligned} \tag{6}$$

where κ<sup>0</sup> = κ(t = 0) and a<sup>0</sup> is the mean activation of the patterns, i.e., a<sup>0</sup> = (NP) <sup>−</sup><sup>1</sup> P<sup>P</sup> µ=1 P<sup>N</sup> i=1 ξ µ i . This definition of the synaptic weights makes the patterns ξ µ i attractors of the activity dynamics of the system, and therefore it constitutes the final step of a process of "learning" or "storing" of a set of activity patterns by the system in the synaptic weights. Notice also that wij = wji by construction so that the network is symmetric, in the spirit of previous studies (Sompolinsky and Kanter, 1986). This is for simplicity and also as a reference to compare with the canonical Amari-Hopfield model (Amari, 1972; Hopfield, 1982).

The overlap of the network state with each of these patterns determines the global state of the system, and it is defined as

$$m^{\mu}(t) = \left[\text{Na}\_0(1 - a\_0)\right]^{-1} \sum\_{i=1}^{N} \left(\xi\_i^{\mu} - a\_0\right) s\_i. \tag{7}$$

It follows from this definition that <sup>−</sup><sup>1</sup> <sup>≤</sup> <sup>m</sup>µ(t) <sup>≤</sup> 1. We say that the system is in a memory state or, equivalently, that it has retrieved pattern <sup>µ</sup>, if <sup>m</sup><sup>µ</sup> <sup>&</sup>gt; <sup>2</sup>/3 and <sup>m</sup><sup>ν</sup> <sup>→</sup> <sup>0</sup> <sup>∀</sup><sup>ν</sup> 6= <sup>µ</sup>. This indicates that the activity state of the network strongly resembles that of pattern µ. In the case of a non-trivial topology, it is also of interest the degree dependent overlap, mµ(k, t), defined as

$$m^{\mu}(k,t) = \left[Np(k,t)a\_0(1-a\_0)\right]^{-1} \sum\_{i=1}^{N} \left(\xi\_i^{\mu} - a\_0\right) s\_i \delta\_{k,k\_i} \tag{8}$$

where p(k, t) is the degree distribution of the network, which indicates the probability that a node has degree k at a certain time t (p(k, t) ≥ 0 ∀k, t, P<sup>N</sup> k=1 <sup>p</sup>(k, <sup>t</sup>) <sup>=</sup> <sup>1</sup> <sup>∀</sup>t). Therefore, <sup>m</sup>µ(t) <sup>=</sup> <sup>P</sup><sup>N</sup> k=1 p(k, t)mµ(k, t). Notice also that if the patterns of activity are not homogeneously distributed through the neurons, mµ(k, t) is not bounded by ±1.

The "canonical" setting of the Amari-Hopfield model, in the case of a fully connected network and random orthogonal patterns, exhibits three characteristic phases. In the absence of thermal noise, T = 0, the patterns ξ µ i are stable attractors of the dynamics of the system for P < P<sup>c</sup> = 0.138N, and the system is in what is called the memory phase. P<sup>c</sup> defines the maximum storage capacity of the network (Amit, 1989), that is, the maximum quantity of information—or number of patterns that can be stored and effectively retrieved from the network. This phase is (mathematically) equivalent to the ferromagnetic or ordered phase of interacting spin networks (as in the Ising model). The storage of a large number of different patterns in the network gives rise to quenched noise as a consequence of the interference between them in wij, which can destabilize such memory phase. Therefore, if P is further increased above P<sup>c</sup> there is a discontinuous phase transition to a spin-glass (SG) phase, in which there appear metastable states which are combinations of the stored patterns (therefore called mixed states) that trap the dynamics of the system. Similarly, in the case of P = 1 when T > T<sup>c</sup> = 1, there is a continuous phase transition from the memory phase to a noisy or paramagnetic phase (also called disordered phase) in which there are no stable attractors, and the dynamics of the system is driven by noise (Amit, 1989). In the more general case in which both T > 0 and P > 1, the location of the phase transitions depends both on T and P. The emergent behavior of the Amari-Hopfield model has also been studied on non-trivial network topologies, such as scale-free and small-world networks (Torres et al., 2004; Boccaletti et al., 2006; Oshima and Odagaki, 2007). Such systems have been shown to present the same phases as the canonical fully connected model, with transition lines that depend on the topology. In particular, it has been reported that, for heterogeneous networks and a single stored pattern, the overlap reduces for T < T<sup>c</sup> , so that memory is recovered but with more errors than in a fully connected network. However, the critical temperature diverges, T<sup>c</sup> → ∞ as N → ∞, due to the presence of hubs that retain pattern information. Therefore, the memory phase expands to much higher values of thermal noise. On the other hand, the capacity of the network is known to decrease as the mean connectivity of the network decreases (Torres et al., 2004).

We here consider an evolving network whose structure, contrary to the canonical model above, changes constantly in time subjected to the pruning dynamics, as we shall describe. Moreover, we consider a highly sparse network, with values of κ/N ∈ - 10−<sup>3</sup> , 10−<sup>2</sup> , which can be homogeneous (i.e., every node having roughly the same connectivity degree), or heterogeneous, with the formation of hubs. Both sparseness and heterogeneity damage severely the memory retrieval ability of the neural network that, for such cases, diminishes fast with P compared with the case of highly connected and homogeneous neural networks (Stauffer et al., 2003; Castillo et al., 2004; Morelli et al., 2004; Torres et al., 2004; Oshima and Odagaki, 2007; Akam and Kullmann, 2014) However, there is experimental evidence that the configurations of neural activity related to particular memories in the animal brain involve many more silent neurons, ξ µ <sup>i</sup> = 0, than active ones, ξ µ <sup>i</sup> = 1 (Chklovskii et al., 2004; Akam and Kullmann, 2014). Notice that in this case there is a positive correlation between different patterns due to the sparseness, since a<sup>0</sup> 6= 0.5, which is also known to improve the storage capacity of a neural network (Knoblauch et al., 2014; Knoblauch and Sommer, 2016), and in particular that of heterogeneous and sparse neural networks (Morelli et al., 2004). Consequently, we consider here this kind of activity patterns, and we further define them as non-overlapping regions of active neurons, each consisting of N/P neurons, so that they cover the whole network (and therefore the mean activity of the patterns is a<sup>0</sup> = P −1 ). This corresponds to a particular definition of sparse or biased patterns, which in other works have been considered to be randomly distributed with a given a<sup>0</sup> (Knoblauch et al., 2014; Knoblauch and Sommer, 2016), what allows for a good visualization of the activity of the network by means of the raster plots.

Moreover, this scheme allows us to define another measure of the overlap between the state of the system and the memorized patterns, considering only the corresponding active neurons as

$$m\_1^{\mu}(t) \equiv \frac{1}{N} \sum\_{i=1}^{N} s\_i(t) \xi\_i^{\mu},\tag{9}$$

with m µ <sup>1</sup> ∈ [0, 1]. If is also of interest its binearized extension, m µ B , defined as m µ B (t) ≡ 1 if m µ 1 (t) > mth and 0 otherwise, so that **mB**(**t**) = m1 B (t), m<sup>2</sup> B (t), ..., m<sup>P</sup> B (t) indicates, in a binary code, which combination of patterns is recovered at time t. Equivalently, the decimal variable d<sup>s</sup> can be defined,

$$d\_s(t) \equiv \sum\_{\mu=1}^{P} 2^{\mu-1} m\_B^{\mu}(t),\tag{10}$$

which one can interpret as a one-dimensional variable indicating the global memory state of the system.

Interestingly, the activity patterns defined here are such that when a number P<sup>r</sup> of them are recovered at the same time, in a SG-like state, the maximum overlap that they can have is less than one. In order to see this, one can decompose Equation (7) in P sums, each over the neurons corresponding to the region associated with each of the activity patterns, as

$$m^{\mu} = \left[\text{Na}\_{0}(1 - a\_{0})\right]^{-1} \sum\_{\upsilon=1}^{P} \sum\_{i=1}^{\mathcal{N}} \left(\xi\_{\mathcal{N}\upsilon - \mathcal{N} + i}^{\mu} - a\_{0}\right) s\_{\mathcal{N}\upsilon - \mathcal{N} + i},\tag{11}$$

where <sup>N</sup> <sup>=</sup> <sup>a</sup>0<sup>N</sup> <sup>=</sup> <sup>N</sup>/<sup>P</sup> is the size of each region and the time dependency has been dropped for clarity. Here, the first sum is over the P patterns stored in the network, whereas the second one goes over the N neurons in the region associated with each pattern. If the pattern µ is recovered together with other P<sup>r</sup> − 1 patterns, then the sum over ν can be split in three terms: the region associated with the pattern µ, the ones corresponding to the other retrieved patterns, and finally those of the non-retrieved patterns (which do not contribute to the sum). Therefore, the overlap corresponding to this pattern is <sup>m</sup><sup>µ</sup> <sup>=</sup> 1 − P −1 −<sup>1</sup> - 1 − P <sup>−</sup><sup>1</sup> <sup>−</sup> (P<sup>r</sup> <sup>−</sup> <sup>1</sup>) <sup>P</sup> −1 . This yields

$$m^{\mu} = 1 - (P\_r - 1) \left(P - 1\right)^{-1} \le 1,\tag{12}$$

which only meets the equality in the case P<sup>r</sup> = 1, that is, if only the pattern µ is retrieved.

The network structure changes in time following a preferential attachment process. This is characterized by the probability each node i has to gain or lose an edge at each time t – namely,

$$\begin{array}{lcl}P\_i^{\mathbb{g}} & = \iota(\kappa)\pi(I\_i),\\P\_i^{l} & = d(\kappa)\eta(I\_i),\end{array} \tag{13}$$

where I<sup>i</sup> = <sup>h</sup><sup>i</sup> <sup>−</sup> <sup>θ</sup><sup>i</sup> is the scaled input that each neuron receives as a consequence of the coupling with its neighbors, a sort of recurrent current in the network, and the time dependence has been dropped for clarity. Here, u and d account for global factors that affect synaptic growth and death, such as the diffusion of different molecules through large areas of tissue, for which the mean degree κ is taken as a proxy. The second terms π and η introduce a dependence on the pre-synaptic activity of the nodes, closing the activity-topology coupling. This creates a feedback loop between the evolution of the structure of the network (form), mediated by the local currents, and the neural activity on the network (function).

Taking the local probabilities to be normalized over the network, the number of edges that are added and removed at each time t depends only on the global probabilities u(κ(t)) and d(κ(t)). In this way, they determine the temporal evolution of the mean connectivity κ(t), whereas the local probabilities π(Ii) and η(Ii) characterize the second order statistics of the network structure, such as the variance of the degree distribution or the degree-degree correlations, as we show below (see also Millán et al., 2018a). These definitions allow us to simulate the dynamics of the system via a Monte Carlo method (in particular, we make use here of the BKL algorithm Bortz et al., 1975) as follows. First, the number of edges to be created and destroyed at time t is sorted according to the global probabilities u(κ(t)) and d(κ(t)). Then, we select as many nodes as indicated with these draws, independently of each other, according to π(Ii) and η(Ii). This process is done in a serial manner, and the same node can be selected more than once. Notice that for each node i that gains or looses an edge eij, the degree of the second node j to which that edge links also changes accordingly. Therefore, there are in fact two paths that can lead to the change of a node's degree: either through the primary process with probability π(Ii) for a gain (or η(Ii) for a loss), or when it is randomly connected to (or disconnected from) an already chosen node. Therefore, the effective values of the second factors in Equation (13) are

$$\begin{array}{lcl}\widetilde{\pi}\_{i} &=& \frac{1}{2} \Big[\pi(I\_{i}) + \frac{1}{N}\Big],\\\widetilde{\eta}\_{i} &=& \frac{1}{2} \Big[\eta(I\_{i}) + \frac{k\_{i}}{\kappa N}\Big],\end{array} \tag{14}$$

where the 1/2 factor is included to assure normalization. Following our previous work, we consider

$$\begin{array}{lcl}\widetilde{\pi}\_{i} &= \frac{I\_{i}^{\alpha}}{\langle I^{\alpha} \rangle N},\\\widetilde{\eta}\_{i} &= \frac{I\_{i}}{\langle I \rangle N},\end{array} \tag{15}$$

which are normalized over the network, P<sup>N</sup> <sup>i</sup>=<sup>1</sup> <sup>e</sup>π<sup>i</sup> <sup>=</sup> P<sup>N</sup> <sup>i</sup>=<sup>1</sup> <sup>e</sup>η<sup>i</sup> <sup>=</sup> 1. The power-law relation in <sup>e</sup>π<sup>i</sup> allows us to explore both suband super-linear responses by just modifying a single parameter, namely <sup>α</sup>. The probabilityeη<sup>i</sup> , on the other hand, is fixed in a linear response, which corresponds to edges being chosen at random for removal, which can be seen as a first order approximation to the pruning dynamics (Millán et al., 2018a). Therefore, α is the control parameter for the pruning dynamics. If α < 1, high degree nodes are more likely to lose edges than to gain new ones, thus creating a homogeneous network structure. On the other hand, if α > 1, high degree nodes are more likely to continue to gain edges than to lose them, which gives rise to a highly heterogeneous, bimodal structure. Finally, the case α = 1 corresponds to the critical case in which networks develop a scale-free topology as shown in previous works (Johnson et al., 2010) that reproduces the scaling behavior observed in the longrange connections of the human brain (Gastner and Ódor, 2016) and in protein interaction networks (Albert, 2005), which decay as a power-law with exponent µ ≈ 2.5.

The local probabilities are then given by

$$\begin{array}{lcl}\pi\left(I\_{i}\right)&=\max\left\{2\frac{I\_{i}^{a}}{\langle I^{a}\rangle N}-\frac{1}{N},0\right\},\\\eta\left(I\_{i},k\_{i}\right)&=\max\left\{2\frac{I\_{i}}{\langle I\rangle N}-\frac{k\_{i}}{\kappa N},0\right\},\end{array} \tag{16}$$

which hold that π(Ii), η(I<sup>i</sup> <sup>P</sup> , <sup>k</sup>i) <sup>&</sup>gt; <sup>0</sup> <sup>∀</sup><sup>i</sup> and normalization, N <sup>i</sup>=<sup>1</sup> <sup>π</sup>(Ii) <sup>=</sup> P<sup>N</sup> i=1 η(I<sup>i</sup> , ki) = 1.

We impose further restrictions on the network. First of all, eij is a binary matrix, so that only one edge per pair of nodes is allowed and the strength of the connection between two neurons, resembling the number of multiple contacts between actual neurons (Fares and Stepanyants, 2009), is considered to be given by wij. Moreover, we set the minimum degree of the network, k<sup>i</sup> = 1, so that there cannot be any disconnected nodes, and we forbid self-connections, eii = 0 ∀i. The maximum degree a node can have is therefore N − 1. We do not impose a hard bound on it as other works have done (Knoblauch et al., 2014; Knoblauch and Sommer, 2016). This would exclusively affect hubs, which only appear for α > 1, as discussed above (Johnson et al., 2010; Millán et al., 2018a), reducing their connectivity. This might affect the memory capabilities of the network in the limit P → ∞ but, since we do not work on this limit, we do not expect any changes on the qualitative behavior and main findings of our model.

Under this framework, the evolution of the mean degree is

$$\frac{d\kappa(t)}{dt} = 2\left[\mu\left(\kappa(t)\right) - d\left(\kappa(t)\right)\right].\tag{17}$$

For a careful derivation of this equation, we direct the reader to Johnson et al. (2010). Intuitively, u κ(t) and d κ(t) set the number of edges that are created and destroyed at every time step, so that u κ(t) − d κ(t) gives the net change in the number of edges. Since for each of these edges two nodes change their degree, there is a factor 2 in the variation of the mean degree. The simplest way to approximate the pruning dynamics is to consider an exponential decay of κ(t) from κ<sup>0</sup> to κ∞, where κ<sup>0</sup> = κ(t = 0) is the initial mean degree of the network and κ<sup>∞</sup> = κ(t → ∞) the stationary mean degree after synaptic pruning has occurred, so that κ<sup>0</sup> ≥ κ∞. This is achieved by defining

$$\begin{array}{l} u(\kappa(t)) = \max\left\{\frac{n}{N}\left(1 - \frac{\kappa(t)}{2\kappa\_{\infty}}\right), 0\right\} \\ d(\kappa(t)) = \frac{n}{N}\frac{\kappa(t)}{2\kappa\_{\infty}}, \end{array} \tag{18}$$

where the parameter n sets the timescale for the pruning dynamics. Notice also that Equation (18) assures that u(κ) ≥ 0 ∀κ. By substituting these definitions into Equation (17), we obtain the time evolution of κ(t),

$$\kappa(t) = \kappa\_{\infty} \left[ 1 - \left( 1 - \kappa\_0 / \kappa\_{\infty} \right) e^{-t/\tau\_p} \right],\tag{19}$$

where τ<sup>p</sup> = Nκ∞/(2n). This set-up has been previously used to reproduce experimental data on the connectivity of the human pre-frontal cortex using values of κ<sup>0</sup> ∈ (60, 80) and κ<sup>∞</sup> ∈ (30, 50), depending on the region, and also of the mouse somatosensory cortex, with κ<sup>0</sup> = 3.10 and κ<sup>∞</sup> = 1.64 (for the other parameters see Johnson et al., 2010; Millán et al., 2018a). The definitions in Equation (18) take into account that synaptic growth and death relay in some way on the concentrations of various molecules (that can have an important role in synaptogenesis, as axonal growth factors), which can diffuse through large areas of tissue and therefore cannot in general be considered local (Klintsova and Greenough, 1999), and here we consider κ(t) as a proxy for the amount of resources consumed by the existing synapses in the network. In an environment with a finite presence of nutrients, it is reasonable to think that there is a competition for the existing resources, and that neurons are sensitive to the amount of nutrients available to them, so that synapses are less likely to grow, and more likely to atrophy, when the connectivity is high, and viceversa, as assumed by Equation (18).

Finally, the network macroscopic state is described via the degree distribution p(k, t) and its homogeneity, defined as

$$\mathbf{g}(t) = \exp(-\sigma^2(t)/\kappa^2(t)),\tag{20}$$

where σ 2 (t) = hk 2 (t)i − κ 2 (t). For homogeneous networks, in which all nodes have similar degrees, σ(t) is small and the homogeneity approaches one, with the trivial case of g = 1 if p(k, t) = δk,k<sup>1</sup> . For heterogeneous networks, on the other hand, there are big fluctuations in the degrees of the nodes and g(t) → 0.

The timescale for structure changes is set by the parameter n, whereas the time unit for activity changes, h<sup>s</sup> , is the number of Monte Carlo Steps (MCS) that the states of all neurons are updated according to the Amari-Hopfield dynamics between each structural network update. Our studies show a low dependence on these parameters in the cases of interest, so we only report here results for h<sup>s</sup> = 10 MCS and n = 10. Measures on the stationary state of the system are carried out by temporal averages of the macroscopic variables, ¯ f = 1 Pt0+<sup>1</sup> t=t<sup>0</sup> f(t).

A recent work (Millán et al., 2018a) showed that, within this framework, three phases emerge: a homogeneous memory phase when both α and T are low (T, α < 1), in which the network is capable of memory retrieval and the topology dynamics keeps a homogeneous configuration; a heterogeneous memory phase for high α (α > 1) in which the dynamics leads to bimodal networks (with the appearance of hubs or highly connected nodes); and a homogeneous noisy phase for high noise T. However, as we will depict in the next section, the combination of thermal noise together with the introduction of a larger number of patterns of activity—which induces interference among them—induces other non-reported non-equilibrium phases characterized by the emergence of complex oscillations among the activity associated with the stored patterns.

#### 3. RESULTS

Previous preliminary analysis of the storage capacity of developing brains under the present framework revealed that the capacity of the network can be greatly improved if a feedback loop between structure and function is considered (Millán et al., 2018a). This is because the interplay between form and function gives rise to a topological structure that enhances the stability of the memory attractors which are recovered during the evolution of the system. In order to explore this interesting picture under other conditions, here we analyze in detail the phase diagram of the system with respect to four relevant parameters in the model, namely, α, κ∞, T, and P. The first two characterize the network structure dynamics, whereas the temperature, T, and the number of stored patterns, P, account respectively for thermal and quenched disorder. As already said, the latter is a consequence of the interference among many stored patterns, and it can affect the recall process. Other parameters, such as the initial connectivity κ<sup>0</sup> or the speed of the pruning, n, where shown to have little or no effect on the dynamics (Millán et al., 2018a,b).

# 3.1. Steady State Solutions for T = 0

We first analyze the behavior of the system at T = 0, that is, in absence of thermal fluctuations that can affect the stability of the fixed point solutions of the system dynamics. As stated above, there are, however, other sources of noise in our system which can have a prominent influence in its behavior. One is the interference among stored patterns, which can significantly reduce the memory retrieval ability of the system (Amit, 1989). Another is the pruning dynamics that adds a second source of noise; this is an intrinsic, structural noise that emerges due to the stochastic adding and removal of synapses associated with the network dynamics during brain development, and which can dynamically affect the performance of the system during memory acquisition and consolidation.

In **Figure 1** we show the corresponding phase diagrams of the system (depicting different phases or kinds of behavior) for different values of κ<sup>∞</sup> = 20, 40, and 60, respectively from left to right. These depict some non-equilibrium phases associated with different computational abilities during memory recall. The top panels show, in the steady state, the ratio of patterns that can be retrieved with high overlap (m<sup>µ</sup> <sup>≥</sup> 0.66), namely <sup>g</sup><sup>P</sup> <sup>≡</sup> <sup>P</sup>r/<sup>P</sup> (where P<sup>r</sup> is the number of retrieved patterns), as a function of α and P. A value g<sup>P</sup> = 1/P indicates a pure memory state, whereas larger values correspond to mixtures and SG-like states (Amit, 1989), and g<sup>P</sup> = 0 corresponds to the noisy or non-memory state. Meanwhile, the middle panels show the mean overlap of the recovered patterns during memory recall, namely m<sup>P</sup> and, finally, the bottom panels show the stationary mean homogeneity, g¯.

These diagrams show up different types of dynamical behavior. In order to illustrate the characteristics of each one, in **Figure 2** we depict the time series mµ(t) (top graph of each panel), raster plots showing the whole activity of the system (bottom graph of each panel) and the steady-state degree distribution (inset of each panel) for some particular values of α and P corresponding to different characteristic behaviors in the phase diagrams in **Figure 1**. For a given stationary connectivity (e.g., κ<sup>∞</sup> = 60, **Figures 1C,F,I**) we find that, for P = 1, the system is able to retain memory for almost every value of α, as it can be seen by the yellow region at P = 1 in **Figure 1F**, which indicates an overlap equal to 1. For small P and small α, SG-like states, or mixture states (in which some of the memories are partially

gP, retrieved after a given transient; the second one (D–F) shows the average overlap with the recovered patterns, mP; and finally the third one (G–I) shows the stationary homogeneity <sup>g</sup>¯. Each column is for a different value of <sup>κ</sup>∞, with <sup>κ</sup><sup>∞</sup> <sup>=</sup> 20, 40, and 60, respectively from left to right. The network size was set here to N = 1, 600 and each point has been averaged over 10 realizations of the system. In this figure a memory phase appears as a blue region in the diagram of g<sup>p</sup> and a high value of mp, indicated by a yellow color. A SG phase appears as a blue region in gp and a lower value of mp, indicated by a green color, whereas a noisy phase appears as black in g<sup>p</sup> and mp. Similarly, homogeneous structures take place for high values of g¯, indicated by a yellow region in the corresponding diagram, whereas heterogeneous structures are for low values of g¯, indicated by a black region. Finally, the pink stars in diagrams (C,F,I) indicate the (P, α) points corresponding to the time series shown in Figure 2.

retrieved at the same time), start to emerge as it is illustrated in **Figure 2A**, which corresponds to the point α = 0.5, P = 10. As a consequence, both g<sup>P</sup> and m<sup>P</sup> take intermediate values; the former since only a finite number of patterns is retrieved, g<sup>p</sup> < 1 (light blue region of the diagram in **Figure 1C**), the later because these are retrieved at the same time, and therefore the overlap is reduced, m<sup>p</sup> < 1 (green and light-blue region of the diagram in **Figure 1F**). In general, however, the observed SG-like states present high values of the overlap with all the recovered patterns due to the high correlation between memories we have considered in this work. Moreover, in this region the network structure is homogeneous since α < 1, so that g¯ approaches 1 and the degree distribution resembles a Poisson distribution (see **Figure 1I** and the inset of **Figure 2A**). In these conditions, when P is increased the memories lose stability until there is a transition from the SG-like state to the noisy one, where the network structure remains homogeneous, as shown in **Figure 2B** for the point α = 0.5 and P = 30. This is indicated by g<sup>P</sup> → 0 (black region in **Figure 1C**), m<sup>P</sup> → 0 (black region in **Figure 1F**), and g¯ → 1 (yellow region on the bottom-right side of **Figure 1I**).

On the other hand, for high α (α > 1), just one (or very few) pattern is retrieved, with m<sup>P</sup> ≈ 1, and the network structure becomes heterogeneous since α > 1 (see inset of **Figure 2C**). As a consequence, g<sup>P</sup> → 1/P, (dark-blue region in **Figure 1C**), m<sup>P</sup> approaches 1 (yellow region in **Figure 1F**), and g¯ → 0 (black region in **Figure 1I**). Memory is achieved due to heterogeneity and the presence of hubs, which can maintain the information content of the retrieved pattern even in the presence of the strong noise induced by the interference with other stored patterns and the dynamic changes of the network structure. Therefore, when P is increased the recovered patterns remain stable, so that m<sup>P</sup> remains close to 1 (**Figure 1F**) and g<sup>P</sup> decreases as 1/P since only one pattern is retrieved (**Figure 1C**, see also the inset of **Figure 2D**, showing the appearance of hubs, and **Figure 1I**, indicating g¯ → 0).

The stationary mean connectivity of the network, κ∞, also affects the behavior of the system, as it determines location of the phase transition from the SG phase to the noisy one for α < 1. As the diagrams in **Figure 1** show, larger values of κ<sup>∞</sup> increase the tolerance of the system to quenched disorder, so that a bigger number of patterns can be stored. This is in line with the known result that the information is stored in the synaptic weights, and therefore increasing the number of synapses also increases the amount of information that the system can store (Amit, 1989).

Notice also that the qualitative state of the system is approximately independent of P for P > 20, as shown in **Figure 1** where one can see that gp, m<sup>p</sup> and g¯ remain essentially constant as P is increased with constant α above P = 20, in agreement with previous studies (Millán et al., 2018a). Therefore, in the following we restrict our analysis to the most interesting region P < 20 and do not analyze the large storage limit of the system (Knoblauch et al., 2014; Knoblauch and Sommer, 2016). This is because our interest here is in characterizing the dynamic behavior arising as a consequence of the interplay between structure and function

phase diagrams of Figure 1. Each composite panel illustrates the overlap time series, mµ(t), (top graph), raster plots of neuron activity (bottom graph) and the steady-state degree distribution of the network (inset), computed at <sup>t</sup> <sup>=</sup> <sup>10</sup><sup>6</sup> Monte Carlo Steps and averaged over 10 realizations of the system. The panels correspond respectively to α = 0.5 and P = 10 (A), α = 0.5 and P = 30 (B), α = 1.5 and P = 10 (C), and α = 1.5 and P = 30 (D). In all presented simulations we set N = 1, 600.

under the presence of thermal and quenched noise, rather than its storage capacity. Similarly, the inclusion of a hard bound on the maximum degree of the nodes would primarily affect the degrees of the hubs of bimodal networks. However, these typically form a highly connected core in the network, so the average path length between nodes would not increase heavily. Therefore, we expect that this bound would not have an important effect in the regime P ≪ N in which we set the system here.

In summary, for T = 0, that is, when there are only two sources of noise in the system (structural and quenched disorder), the stationary state for a given P depends strongly on the network structure, determined by α. As so, for α > 1, the network develops heterogeneous structures in which hubs arise. These are very densely connected with the rest of the network, and can maintain information about the memories even when P is very high. For α < 1, on the other hand, the network is always homogeneous, with every node having similar, low degree, and a SG-like phase soon arises, which is then suddenly lost as the quenched disorder becomes too strong and finally the system falls into the noisy state.

# 3.2. Behavior of the System for T > 0

Our previous analysis has determined the phase diagram of the system at T = 0, which characterizes the effect of the dynamical topological structure on the memory capabilities of the system. In this section, we consider the effect of thermal noise in our system's emergent behavior. In order to do so, we analyze in **Figure 3** the phase diagrams of the system with respect to α and T, for some representative numbers of stored patterns, namely P = 5, 10, 15, and 20 (each column of the figure corresponds to a different P), and for three values of κ<sup>∞</sup> = 20, 40, and 60, as before. The selected values of P correspond to the left region of diagrams in **Figure 1**, where the phase transitions from memory to the SG and noisy states takes place. In order to illustrate better the behavior of the system in the cases of interest, we also include in **Figure 4** the time series of mµ(t) is some exemplary

FIGURE 3 | Phase diagrams of the system with respect to α and T for four different values of P, in particular for P = 5, 10, 15, and 20, respectively from left to right, and for three values of <sup>κ</sup><sup>∞</sup> <sup>=</sup> 20, 40, 60, respectively from top to bottom (A–L). In each panel we show three diagrams: <sup>g</sup>p, <sup>m</sup>p, and <sup>g</sup>¯, as indicated in the label of the color bar. Pink stars in (A–D) indicate the (T, α) point of the corresponding time series in Figure 4. Results are for N = 1, 600 and have been averaged over 5 realizations of the system dynamics. In this figure a memory phase appears as a blue region in the diagram of gp and a high value of mp, indicated by a yellow or green color. A SG phase appears as an orange region in gp and a lower value of mp, indicated by a green or blue color, whereas a noisy phase appears as black in gp and dark-blue in mp. Finally, the oscillatory phase appears for high values of gp, (light yellow regions in the corresponding diagrams) and relatively low values of mp (associated blue regions of the corresponding diagrams). Similarly, homogeneous structures take place for high values of g¯, indicated by a yellow region in the corresponding diagram, whereas heterogeneous structures are for low values of g¯, indicated by a black or dark blue region.

points for κ<sup>∞</sup> = 20, as indicated in the phase diagrams by a pink star. Each panel corresponds to a given value of P, and each graph on them to a point in the (T, α) space. We find that the combination of thermal and quenched disorder, associated with the interference among patterns, can give rise to oscillations among the memorized patterns for α < 1—that is, when the networks are homogeneous—and T < 1, which are correspond to the yellow regions in the g<sup>p</sup> panels of **Figure 3**. Note that the observed oscillations occur at level of the neuronal population as measured by the global network parameter mµ(t), and not on the single neuron level—which appear as small, high-frequency oscillations of mµ(t).

In order to illustrate the emergent behavior of the system, we refer here to **Figure 3A**, which corresponds to κ<sup>∞</sup> = 20 and P = 5. The top graph in the panel represents gP, the number of patterns visited by the system after the transient

evolution takes place. For α > 1, we find that this number remains finite, and greater than zero, up to very high values of the temperature (T ≈ 2.0) corresponding to the lightblue region in the top panel of **Figure 3A**, for instance. This indicates that the system is in a memory state (or in a SG-like state in which only a small number of patterns are retrieved), such as the ones depicted in the bottom graphs of **Figure 4A**. The stability of the memory state for T > 1 is possible due to the emergence of heterogeneous structures (since α > 1), and consequently hubs, as indicated in the black region of the bottom plot of **Figure 3A**, since g¯ → 0 for α > 1. Notice also in the middle graph of **Figure 3A** how, as T is increased, the overlap corresponding to these states, mP, decreases, indicating that these states are also becoming less stable as the thermal noise becomes stronger. In these conditions, only the more densely connected hub nodes are able to maintain information about the memories, and these are the ones contributing the most to the overlap.

As α is decreased, however, the behavior shown in the diagrams becomes more complex and different regions (phases) start to emerge. We find, as expected, that memory is completely lost for T ≫ 1, i.e., due to the strong noise the system falls into the noisy or non-memory state (as indicated respectively by the black region and by the dark-blue region of the top and middle diagrams of **Figure 3A**). In fact, now networks are homogeneous (α < 1) and there are no hubs that preserve memory (as indicated by g¯ → 1 in the bottom plot of **Figure 3A**, indicating that the degree distribution is homogeneous). A typical time series of mµ(t) for this situation is shown in the top-right graph of **Figure 4B**.

For small values of T and α (T, α < 1) on the other hand, g<sup>P</sup> → 1 (orange and yellow region of the top panel of **Figure 3A**), indicating that a great number of patterns are being retrieved (P<sup>r</sup> → P) with a moderate value of the overlap mP, as indicated by the green and blue region in the middle panel of **Figure 3A**. Moreover, results in **Figure 2** indicate that, at least in some cases, g<sup>P</sup> actually increases when T goes from 0 to 1 (see, e.g., the yellow area in the top panel of **Figure 3A**). That is, as the temperature increases, more memories take place in the state of the system, with a relatively high overlap mP. This is because, for α < 1, there is a wide region of oscillatory behavior between the SG-like and the noisy phases corresponding to the yellow region of the g<sup>p</sup> diagrams in **Figure 3**. An exemplary series of oscillations is illustrated in the top-left graph of **Figure 4A**. This emerges as a consequence of the interplay between structural and thermal noise and the activity of network, since the process of addition and removal of synapses, that creates a dynamical network structure, together with the thermal noise, can make the recovered patterns unstable. Moreover, given that α < 1, the structure of the networks remains homogeneous and no real hubs emerge. Notice however that due to the non-trivial interplay between activity and topology, in the region of oscillatory behavior the networks display a more heterogeneous structure, and g¯ < 1. This effect will be discussed in more detail in the following section.

As in the previous section, we have also analyzed the role of the stationary mean connectivity, κ∞, on the phase diagram of the system for T > 0 (see **Figure 3**). This parameter holds physiological interests since it can be taken as a measure of the extension of the process of synaptic pruning. From this point of view, a brain that has undergone a more drastic synaptic pruning would have smaller κ<sup>∞</sup> than one that has been less pruned. This is to be related to recent experiments that have associated an excessive pruning in certain brain areas with schizophrenia (Sekar et al., 2016), whereas ASD has been related to a defect of synaptic pruning (Tang et al., 2014). We find in our model that the area associated with the oscillatory behavior (for α, T < 1) for a given κ<sup>∞</sup> is maximum at intermediate values of P: for very small P there is a dominance of stable SG-like states, whereas for large P the system falls easily on the noisy phase. Similarly, for a given P the greater extension of the oscillatory phase is found for an intermediate κ∞. For instance, for P = 10 the noisy phase extends to T < 1 for κ<sup>∞</sup> = 20 (**Figure 3B**) and the oscillatory region is small, whereas for κ<sup>∞</sup> = 60 (**Figure 3J**) there is a combination of stable SG-like states and oscillations for α, T < 1. Finally for κ<sup>∞</sup> = 40 (**Figure 3F**) the oscillatory phase is most robust. Consequently, the absence of dynamical memories in the system could be associated with a defect of the pruning process that causes κ<sup>∞</sup> to be greater than usual, and could be therefore associated with ASD. Interestingly, it has been recently reported that short-term memory and episodic memory are impaired in ASD subjects (Poirier et al., 2011; Lind et al., 2014), which is consistent with our findings here since, in order to be able to recall a sequence of memories, it is first necessary to destabilize the already recalled ones so as to allow the system to remember new ones. On the other hand, schizophrenia is typically associated with erratic behavior (Loh et al., 2007), which could be related to the high frequency memory oscillations found here for smaller values of κ∞.

### 3.3. Emergence of Hubs

The appearance of hubs and heterogeneity plays a significant role in the emergent dynamics of the system. In particular, with a given level of noise (T > 0), the topological structure of the network determines whether the system relaxes to a memory state, wanders among different patterns or falls into a noisy state. Therefore, here we discuss in more detail the emergence of hubs during the network evolution and their effect on the emergent state of the system.

We first notice that, according to the previous analysis, for α < 1 networks are homogeneous, as evidenced by the homogeneous degree distributions shown in the insets of **Figures 2A,B**. This is also revealed by the high value of the homogeneity parameter g¯ shown in **Figures 1**, **3** for α < 1, indicating that the variance of k<sup>i</sup> is small. As a consequence, no real hubs can be defined, since all nodes have similar low degree (given that κ<sup>∞</sup> ≪ N, so that the connectivity of the nodes is bounded). On the contrary, for α > 1 and in the case of memory, networks are heterogeneous as evidenced by g¯ → 0 (black regions in the corresponding diagrams g¯(α, P) and g¯(α, T) respectively in **Figures 1**, **3**). This indicates that there are nodes with very different degrees and, in particular, the degree distribution p(k, t) is bimodal and it splits in two, as shown in the insets of **Figure 2C,D**, with the emergence of hubs. Therefore, one can set the connectivity threshold kth—that defines the minimum node's degree to characterize it as a hub—at the value of k at which p(k, t → ∞) presents a local minimum between the two modes. This establishes a clear separation between high and low degree nodes. In particular, in all cases studied here, we find that a threshold kth = 2κ<sup>∞</sup> also suffices to differentiate between homogeneous and heterogeneous structures, since for α < 1 (homogeneous case) the maximum degree of a network is always below 2κ∞.

Interestingly, due to the underlying stochastic rewiring process and to the system's finite size, there is always some variability in the degrees of the nodes and, particularly in the region of oscillatory behavior, there is a relative increase in the variability of k<sup>i</sup> with respect to the SG phase (as evidence by a decreased g¯ in the corresponding diagrams of **Figure 3**). We argue that this is due to the intrinsic coupling between activity and topology, and to the combination of thermal (since T > 0), topological (due to the ongoing rewiring process) and quenched (due to the learning of different patterns) disorder in the system. In the region of oscillatory behavior, the instability of the memories influences the synaptic currents I<sup>i</sup> creating variability, thus causing the observed increased heterogeneity. This causes the emergence of relatively-high degree nodes that correspond to the tail of the homogeneous distribution p(k, t → ∞) and which might have an important effect on the system. Therefore, in order to explore as well the dynamics of these relatively-high degree nodes, we have selected a lower threshold, kth = 1.75κ∞, for the analysis of hub dynamics.

Hubs (and relatively-high degree nodes for the homogeneous case) dynamics is investigated in **Figure 5**, where we compare two different cases for P = 5 and κ<sup>∞</sup> = 20. The first one, shown in **Figures 5A,B**, corresponds to the region of oscillatory behavior for homogeneous networks (α < 1) and it is for T = 0.7 and α = 0.3, corresponding to the bottom-left graph of **Figure 4A**. The second one, shown in **Figures 4C,D**, corresponds to the heterogeneous-memory phase for α > 1, and it is for T = 0.3 and α = 1.1, corresponding to the top-left graph of **Figure 4A**. In particular, we analyze in **Figures 4A,C** the temporal evolution of the system as given by the overlap mµ(t), the homogeneity g(t) and the hub raster plots, where we represent the existing hubs at each time t, in different colors according to their active or inactive state (respectively pink and green). Furthermore, in **Figures 4B,D** we show the degree-dependent overlap mµ(k, t0) [defined in Equation (8)] and the degree histogram N(k, t0) = Np(k, <sup>t</sup>0), for a particular time, <sup>t</sup><sup>0</sup> <sup>=</sup> <sup>5</sup> · <sup>10</sup>6MCS, corresponding to the systems respectively depicted in **Figures 4A,C**.

We observe, for α > 1 (**Figure 5C**), that a great number of hubs emerge in the system, and that almost all hubs correspond to the active nodes of the retrieved pattern. Moreover, in this case mµ(k, t0) of the recovered pattern µ is larger for highdegree nodes (**Figure 5D**), indicating that they contribute most to the overlap mµ(t0). On the contrary, for the non-recovered patterns ν, m<sup>ν</sup> (k, t0) remains small for all k. On the other hand, for α < 1, no real hubs emerge and only transient relativelyhigh degree nodes are observed in **Figure 5A**. These do not

only correspond to the recovered patterns but are scattered throughout the network, and no significant correlation can be measured between the pattern oscillations and the hubs dynamics. This causes instabilities that ultimately lead to the oscillatory behavior (see **Figure 5B**, indicating that relatively high-degree nodes contribute more to mµ(k, t) of the recovered patterns but not only).

In summary, **Figure 5** shows that for α > 1 there are active hubs in the system that correspond to the recovered pattern, making it stable. On the other hand, for α < 1 no real hubs can emerge in the system, and the transient relativelyhigh degree nodes are scattered throughout the network, not only corresponding to the recovered pattern, thus inducing the observed oscillatory behavior.

#### 3.4. Analysis of the Oscillatory Behavior

In the previous sections we have shown the emergence of oscillations for α < 1 and T > 0 and their relation to the existence of transient relatively-high degree nodes on the network. Here, we develop further on the structure and patterns of these oscillations. For simplicity, we focus on the case of κ<sup>∞</sup> = 20 and P = 5 as before, and in **Figure 6** we show a long time series corresponding to this oscillatory phase (T = 0.7 and α = 0.3 as in the top graph of **Figure 4A** and in **Figure 5A**). Plots of the active-overlap parameter m µ 1 (t) (**Figure 6A**) defined in Equation (9), its binearized version m µ B (t) (**Figure 6B**) and the global memory state parameter ds(t) defined in Equation (10) (**Figure 6C**) indicate that the state of the system corresponds to oscillations between SG-like states in which either 2 or 3 patterns are transiently retrieved. These plots also evidence that the oscillations do not follow any clear periodic or regular pattern.

In order to analyze the pattern of oscillations, we show the power spectra of m µ B (t), Sµ(f), and of ds(t), Ss(f), in **Figure 6F**, which displays a power-law decay with an exponent equal to −0.9, indicating that there is not a dominant frequency of the oscillations, but that jumps between different patterns occur at all time scales. This is in accordance with previous studies that have repeatedly reported 1/f-type noise in brain activity under healthy conditions. It has been reported, for instance, in electroencephalogram (EEG) and functional magnetic resonance (fMRI) measures of human brain activity (Linkenkaer-Hansen et al., 2001; Voytek et al., 2015) and also in behavioral processes related to human cognition and motion as well as animal motion (Chialvo, 2010). 1/f noise indicates the existence of temporal correlations within the data, and has been related to emerging self-organized criticality in the brain (Chialvo, 2010).

Moreover, we also investigate the frequency of appearance of each global state, as seen in **Figure 6E**, which indicates that global states have different probabilities of occurrence in each realization of the system. However, when averaged over realizations, the mean probability of each state, p¯<sup>s</sup> , converges to a uniform distribution p¯<sup>s</sup> → 1/N<sup>s</sup> , where N<sup>s</sup> = 20 is the total

number of possible 2− and 3−pattern states (p¯<sup>s</sup> = 0.054(3) when averaged over 20 realizations). Similarly, individual patterns may have different probability of appearance, pµ, in each realization of the dynamics, but when averaged over realizations p¯<sup>µ</sup> converges to 1/P [p¯<sup>µ</sup> = 0.21(1)]. Finally, we also computed the transition matrix between global SG-like states (**Figure 6F**) which indicates that, in a given realization of the system, some transitions are preferred by the system depending on the emergent coupling between activity and topology.

In summary, these results show that the oscillations are not periodic, but occur at all time scales, and that all SGlike states are visited in a non-periodic order. Interestingly however, in a given realization of the system not all transitions are allowed, but only some of them occur. It could be interesting to analyze in more detail in further studies whether the coupling between structure and activity induces a particular pattern of oscillations, and how the scaling of the frequency of oscillations depends on the parameters of model.

# 4. DISCUSSION

We report here on recent studies of the emergent behavior of developing brain models in which structure and function cooperate and influence each other through a feedback loop, thus affecting the system's memory storage and retrieval abilities. This is a prominent example of how inter-synaptic factors at the network level can affect the processing of information in developing brains in a nontrivial way. In particular, our study focuses on the analysis of the conditions under which such feedback loop can enhance the storage and retrieval of a set of correlated patterns. Our work also pays attention to the emerging dynamics of the system, which is a consequence of the interplay between structural, quenched, and thermal disorder during its maturation. The results presented here demonstrate that a heterogeneous network can greatly improve the stability of the memory patterns, since its structure is optimized to preserve information about them in the network hubs which, as we have shown, correspond to the active neurons of the retrieved memories during the recall process. Moreover, due to the structural plasticity, once a pattern is retrieved, the ability of the system to recall it again increases thanks to pruning optimization. This illustrates the constructive role of synaptic pruning to consolidate memories in the "memory phase" of the system.

Our study also shows that the interplay between thermal noise, the interference among stored patterns and the dynamics driving the evolution of the topology creates instabilities on the memory attractors, which can make the system wander among different configurations for certain values of the parameters. In our model, these oscillations among stored patterns are caused by the topological synaptic plasticity due to the death and birth of synapses, which change the energy landscape of the system. In fact, in the absence of this rewiring process, this oscillatory phase is not present and the model would reduce to an Amari-Hopfield model on top of a non-trivial fixed topology. This has been shown to present the same phases as the canonical fully connected version of the model, with transition lines that depend on the topology, so that for instance the critical temperature diverges, T<sup>c</sup> → ∞ as N → ∞, due to the presence of hubs that retain pattern information. Interestingly, in our model the oscillatory behavior that takes place on the homogeneous networks phase of the system is also associated with an increased transient heterogeneity of the underlying structure, in which transient relatively-high degree nodes (whose degree is however smaller than typical hubs) emerge and disappear in time. Moreover, these relatively-high degree nodes do not correspond in general to the active nodes of the transiently recovered patterns, but appear distributed throughout the whole network, corresponding also to active nodes of the rest of non retrieved patterns. This creates a non-trivial time-dependent competition among the different patterns which, together with the subsequent removal of some synapses during brain development, can make the currently recalled attractor less stable, thus inducing the observed wandering among the memories.

We have also analyzed the characteristics of the oscillations and shown that the oscillatory pattern is not periodic but presents a power spectrum following a power law scaling decay with an exponent of −0.9, so there are not any preferred frequencies. This in accordance with previous studies repeatedly reporting 1/f noise in brain activity under healthy conditions (Linkenkaer-Hansen et al., 2001; Voytek et al., 2015) and also in behavioral processes (Chialvo, 2010), which is related to the existence of temporal correlations within the data, and has been related to emerging self-organized criticality (SOC) in the brain (Chialvo, 2010).

Interestingly, the appearance of an oscillatory phase characterized by dynamical memories could be useful to enhance the learning and recalling of sequences of patterns of activity, as in episodic memories, without the necessity of any external input or current forcing the retrieval of the memories in the sequence. This type of oscillations has already been reported for brain models with synapses enduring short-term synaptic plasticity (STP) (Pantic et al., 2002; Cortés et al., 2006; Marro et al., 2007; Torres et al., 2007, 2008). This occurs at the synapse level and depends on the activity of the pre-synaptic neuron (which therefore depends closely on the synaptic current I<sup>i</sup>

used in our model Amit, 1989). However, STP is caused by biophysical mechanisms controlling the release and recycling of neurotransmitters at the synapses during synaptic transmission and operates at short time scales of the order of ms (Tsodyks and Markram, 1997). The activity dependent topological plasticity reported here, however, is the result of the interplay between form and function in a developing brain, and the ongoing synaptic rewiring in mature brains, which happens at the time scale of hours or days (Holtmaat and Svoboda, 2009). Moreover, topological plasticity allows the system to explore more efficiently its dynamical phase space and it has been shown to improve the capacity of neural networks by allowing them to organize in a more efficient structure. Both mechanisms could happen at the same time in actual systems, together with neuron level phenomena such as spike adaptation (Knoblauch and Palm, 2002; Ha and Cheong, 2017). We hypothesize that the combination of these mechanisms could lead to the extension of the oscillatory behavior to other regions of the phase diagram, although results would strongly depend on the relative time scale between structural plasticity and STP, and it could be an interesting approach for future works.

It is also worth noting that the reported oscillations in our system are for the overlap function that is a measure of the activity of the whole neuron population during memory recall processes. These occur in actual neural systems at a long time scale—normally days or even years—as it is the case in our model. Temporal changes at the single neuron level appear in our system as high frequency fluctuations in the time dependent value of the overlap parameter. If the level of stochasticity is low (low T) and the network size is large enough (N ≫ 1), such single neuron fluctuations are very unlike to be significant on mµ(t). In any case, the model output could be easily tuned up to obtain faster of slower oscillations in the overlap function to match more realistically actual experiments during learning and recalling. This could be done by varying some model parameters to make the recall process more or less efficient in time, or to allow the system to recall dynamic memories—such as episodic memories—that are learned and recalled at different stimuli input frequencies.

We have analyzed in detail how the dynamical behavior of the system depends on the synaptic factors affecting the addition and removal of synapses and on the number of stored patterns. In particular, the stationary mean connectivity of the network, κ∞, has been shown to have a great effect on the emergent behavior of the system. For instance, we have found that the absence of dynamical memories in the system, or the presence of memory oscillations with long periods, is associated with a defect of the pruning process. Similarly, we have shown that high frequency oscillations among patterns and more tendency to noisy behavior occur when there is a pruning excess. In particular, the destabilization of recovered memories is necessary for instance to recall a sequence of memories, each during a short period of time, so as to allow the system to remember new ones. One may argue that the induced instability and the associated oscillatory behavior observed in our system could be positive for information processing, since it would allow neuronal media to explore different memories or attractors, for instance following hetero-clinic orbits, and consequently, to process more complex

information, such as spatio-temporal patterns of information (see Rabinovich et al., 2006 and references therein). Such emergent behavior could also be useful to respond more efficiently to changing external stimuli, as in episodic memory tasks, as it has been widely stated in previous works in different neural systems (Cortés et al., 2006; Marro et al., 2008; Torres et al., 2008).

The different emergent behavior in the model with varying connectivity could perhaps be associated with cognitive abilities related to autism spectrum disorders (ASD) and schizophrenia. In the former, a pruning defect has been observed in some brain areas (Tang et al., 2014), and it has recently been reported that short-term memory and episodic memory are impaired in ASD subjects (Poirier et al., 2011; Lind et al., 2014). Our results are consistent with these observations since, if the brain is less pruned, the mean connectivity of the corresponding network is higher, thus making the memory attractors more stable. This implies a lower ability of the brain to remember sequences of patterns as described in episodic memory tasks in ASD patients because it is harder for the brain to forget the already recalled pattern due to its strong stability. Results in our model indicate that a lightly pruned brain could be forced out of the memory phase into an oscillatory regime with an increase in the number of stored patterns (see **Figure 3**). These observations might provide an interesting insight for experimental psychologists to design a cognitive strategy or therapy to learn and recall sequences of patterns, that might improve the cognitive abilities of patients with ASD. On the other hand, we have demonstrated that high frequency oscillations among patterns occur when there is a pruning excess, and this could be perhaps associated with the erratic behavior observed in schizophrenia (Loh et al., 2007), in which case the brain seems to present some areas with an excess of pruning (Sekar et al., 2016). In this case our results here suggest that a learning therapy based on increasing the number of stored memories would not be useful but may in fact be detrimental, as it would make the memory activity patterns more unstable. A learning therapy that moves the patient brain state near to its stable memory phase, for instance, by stabilizing a few old useful memories, could therefore be more convenient.

Finally, we note also that some drastic assumptions have been made in order to simplify the relevant scenario. Firstly, our study is for sparse correlated patterns, as suggested by experimental studies (Chklovskii et al., 2004; Akam and Kullmann, 2014), which are also known to improve the memory retrieval capabilities of the network (Knoblauch et al., 2014; Knoblauch and Sommer, 2016) and particularly so in the case of highly sparse and heterogeneous networks (Morelli et al., 2004). Moreover, we have selected the patterns of activity to be non-overlapping regions of activity, following previous works (Torres and Marro, 2015). This set up corresponds to a particular case that allows for a better visualization of the network dynamics and that has proven out to be useful to investigate the interplay between structure and dynamics, i.e., between form and function, together with the presence of thermal and quenched disorder, on a developing neural network. Similarly, results are for the low storage regime of the neural network, P ≪ N, what allows us to study in detail the dynamical behavior of the system that gives rise to memory wandering. However, given that our qualitative results depend little on P for P > 20, we expect them to hold when P is increased.

Further extensions of this work could also include the consideration of different details of the synaptic pruning process, including for instance the growth of synapses taking place after birth (Millán et al., 2018b), multiple synaptic contacts between neurons (Knoblauch et al., 2014; Knoblauch and Sommer, 2016), or a hard bound on the maximum degree of the nodes (Stepanyants et al., 2002; Fares and Stepanyants, 2009). Moreover, more elaborated definitions of the probabilities of growth and death of synapses (Equations 16 and 18) could also be considered, such as a mechanism of self-organization toward the stationary mean connectivity (Chechik et al., 1999; de Arcangelis et al., 2006; Lewis and Todd, 2007; Tetzlaff et al., 2010) or by explicitly including a dynamics for the available nutrients (Tetzlaff et al., 2010). However these definitions would still need to reproduce the basic characteristics of brain development and synaptic pruning, that is, an initial fast decay of connectivity and an ongoing rewiring of edges after the stationary mean connectivity has been reached. We expect that our main results (existence of a feed-back loop between structure and activity, bistability, and emergence of oscillations) would still hold, at least qualitatively, with these modifications, in accordance with previous studies (Millán et al., 2018b). Similarly, the local probabilities in Equation (16) could also consider more detailed functions, to characterize for instance a specific dependence on the concentration of different proteins and growth factors controlling synaptic growth. These could be obtained experimentally, although to the best of our knowledge it has not yet been done. Our work could thus motivate neurobiologists to design experiments to describe the exact probabilities involved in synaptogenesis and pruning, information that could be easily incorporated in our theoretical framework. Similarly, it could also be particularly interesting to include a learning dynamics that is also coupled to the development of the neural network, thus modeling learning during infancy, or to include an external current on the system, that could certainly be time dependent, to analyze the effect of external inputs on associative memory and memory wandering.

# AUTHOR CONTRIBUTIONS

APM, JJT and JM designed the analyses, discussed the results, and wrote the manuscript. APM also wrote the codes and performed the simulations.

# FUNDING

We are grateful for financial support from the Spanish Ministry of Science and Technology, and the Agencia Española de Investigación (AEI) under grant FIS2017-84256-P (FEDER funds) and from the Obra Social La Caixa (ID 100010434, with code LCF/BQ/ES15/10360004). This study has been also partially financed by the Consejería de Conocimiento, Investigación y Universidad, Junta de Andalucía and European Regional Development Fund (ERDF), with reference SOMM17/6105/UGR.

### REFERENCES


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Millán, Torres and Marro. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.