Neuromorphic Silicon Neuron Circuits

Hardware implementations of spiking neurons can be extremely useful for a large variety of applications, ranging from high-speed modeling of large-scale neural systems to real-time behaving systems, to bidirectional brain–machine interfaces. The specific circuit solutions used to implement silicon neurons depend on the application requirements. In this paper we describe the most common building blocks and techniques used to implement these circuits, and present an overview of a wide range of neuromorphic silicon neurons, which implement different computational models, ranging from biophysically realistic and conductance-based Hodgkin–Huxley models to bi-dimensional generalized adaptive integrate and fire models. We compare the different design methodologies used for each silicon neuron design described, and demonstrate their features with experimental results, measured from a wide range of fabricated VLSI chips.

of the physics of neural computation that are fundamentally different from digital principles in traditional computing, initiated the investigations in the field of neuromorphic engineering (Mead, 1989). Silicon neurons (SiNs) are hybrid analog/digital very large scale integration (VLSI) circuits that emulate the electrophysiological behavior of real neurons and conductances. Hardware emulations of neural systems that use SiNs operate in real-time, and the speed of the network is independent of the number of neurons or their coupling. SiNs offer a medium in which neuronal networks can be emulated directly in hardware rather than simply simulated on a general purpose computer. They are much more energy efficient than simulations executed on general purpose computers, so they are suitable for real-time large-scale neural emulations (Silver et al., 2007;Schemmel et al., 2008). On the other hand, SiN circuits provide only a qualitative approximation to the exact performance of digitally simulated neurons, so they are not ideal for detailed quantitative investigations. Where SiN circuits provide a tangible advantage is in the investigation of questions concerning the strict real-time interaction of the system with its environment (Indiveri, 2000;Le Masson et al., 2002;Vogelstein et al., 2008;Indiveri et al., 2009;Mitra et al., 2009). And the technology developed to build

IntroductIon
Spike-based models of neurons have recently become very popular, for both investigating the role of spike-timing in the computational neuroscience field, and for implementing event-driven computing systems in the neuromorphic engineering field. Several spike-based neural network simulators have been developed within this context, and much research has focused on software tools and strategies for simulating spiking neural networks (Brette et al., 2007). Digital tools and simulators are convenient and practical for exploring the quantitative behavior of neural networks. However they are not ideal for implementing real-time behaving systems, or detailed largescale simulations of neural systems. Even the largest supercomputing systems to date are not capable of obtaining real-time performance when running simulations large enough to accommodate multiple cortical areas, yet detailed enough to include distinct cellular properties. Custom digital systems that exploit parallel graphical processing units (GPUs) or field programmable gate arrays (FPGAs) may offer such capabilities in due time, but it is not clear that such systems will be able to approach the density, energy efficiency, and resilience of neurons and synapses that they model in the central nervous system. The observation that the brain operates on analog principles one or more of the following stages: A (linear or non-linear) temporal integration block, a spike generation block, a refractory period block and a spike-frequency or spiking threshold adaptation block. Each of these functional sub-blocks can be implemented using different circuit design techniques and styles. Depending on which functional blocks are used, and how they are combined, the resulting SiN can implement a wide range of neuron models, from simple linear-threshold units to complex multi-compartmental models.
The dendrites and axon circuit blocks can be used to implement the cable equation, for modeling signal propagation along passive neuronal fibers (Koch, 1999). These circuits allow the design of multi-compartment neuron models that take into account the neuron spatial structure. We will describe examples of such circuits in Section 3.5.
Design styles Table 1 summarizes the relevant computational sub-blocks useful for building SiNs and the possible design styles that can be used to implement them. Each computational block can be implemented with circuits that adopt any of the design strategies outlined in the bottom part of the table. The terms weak and strong inversion in that table refer to the region of operation of individual MOSFETs: In the weak-inversion (or sub-threshold) region the transistor current flow mechanism is diffusion, while in the strong-inversion (or above threshold) region, it is drift. The voltage-mode and current-mode design styles refer to the way input and output signals are represented (i.e., with voltages or currents respectively). S-C designs implement discrete time signal processing strategies, by using clocked switches (MOSFETs) to move charge from one capacitor to the next. Conversely in nonclocked systems, signals are continuous and no global clock circuit is necessary. Biophysical and phenomenological models refer to the level of detail used in the SiN circuit, to implement a model of a real neuron. And the last two design styles, real-and accelerated-time refer to the range of time scales that can be emulated in hardware. Circuits that can operate with time-constants that are biologically plausible are said to be real-time, while circuits that can only run at time scales which are a factor to 10 or more faster, are said to be accelerated-time.
In the next Section we will describe some of the more common circuits used as basic building blocks for building SiNs which cover all design strategies outlined in Table 1. these real-time, low-power neuromorphic systems can be used to engineer brain-inspired computational solutions for practical applications. The term "neuromorphic" was coined by Carver Mead in the late 'eighties to refer to artificial neural systems whose architecture and design principles are based on those of biological nervous systems (Mead, 1990). SiN circuits represent therefore one of the main building blocks for implementing neuromorphic systems. Although in the original definition, the term neuromorphic was restricted to the set of analog VLSI circuits that operate using the same physics of computation used by the nervous system (e.g., silicon neuron circuits that exploit the physics of the silicon medium to directly reproduce the bio-physics of nervous cells), the definition has now been broadened to include analog/digital hardware implementations of neural processing systems, as well as spike-based sensory processing systems. Within this context, many different types of SiNs have been proposed, that emulate real neurons at many different levels: From complex biophysical models that emulate ion channel dynamics and detailed dendritic or axonal morphologies to basic integrate-and-fire (I&F) circuits. Depending on the application domain of interest, SiN circuits can be more or less complex, with large arrays of neurons all integrated on the same chip, or single neurons implemented on a single chip, or with some elements of the neuron distributed across multiple chips.
In this work we describe a wide range of circuits commonly used to design SiNs, spanning multiple design strategies and techniques that range from current-mode, sub-threshold to voltagemode, switched-capacitor (S-C) designs. Moreover we present an overview of the most representative silicon neuron circuit designs recently proposed, compare the different approaches followed, and point out advantages and strengths of each design.

SIlIcon neuron computatIonal blockS
From the functional point of view, silicon neurons can all be described as circuits that have one or more synapse blocks, responsible for receiving spikes from other neurons, integrating them over time and converting them into currents, as well as a soma block, responsible for the spatio-temporal integration of the input signals and generation of the output analog action potentials and/or digital spike events. In addition both synapse and soma blocks can be interfaced to circuits that model the neuron's spatial structure and implement the signal processing that takes place in dendritic trees and axons respectively.
The synapse circuits of a SiN can carry out linear and non-linear integration of the input spikes, with elaborate temporal dynamics, and short and long-term plasticity mechanisms. The temporal integration circuits of silicon synapses, as well as those responsible for converting voltage spikes into excitatory or inhibitory post-synaptic currents (EPSCs or IPSCs respectively) share many common elements with those used in the soma integration and adaptation blocks. Therefore in this paper we restrict our analysis of synapse circuits only to those circuits that implement the basic functionalities of voltage-spike to current conversion and temporal integration, while their complex non-linear features and their spike-timing dependent plasticity mechanisms will be the focus of a subsequent paper.
The soma block of a SiN can be further subdivided into several functional blocks that reflect the computational properties of the theoretical models they implement. Typically SiNs comprise (spikes) arriving at the V in node are integrated to produce an output current I syn with exponential rise and decay temporal dynamics. The circuit time-constant can be set by adjusting the V t bias, and the maximum current amplitude (e.g., corresponding to synaptic efficacy) depends on both V t and V w . A recent current-mode circuit that implements temporal dynamics using this log-domain LPF circuit coupled to a wide-range transconductance amplifier has been proposed in Rachmuth and Poon (2008). This circuit allows robust emulation of emergent iono-neuronal dynamics, reproducing also chaotic bursting as observed in pacemaker cells. A detailed analysis of the synaptic and neural dynamics that can be obtained with the log-domain LPF circuit is presented in Bartolozzi and Indiveri (2007). In Bartolozzi and Indiveri (2007) the authors propose also additional circuits for implementing synaptic dynamics, including a novel differential pair integrator (DPI) circuit (see Figure 1C). Similar to the LPF pulse integrator, the DPI circuit integrates voltage pulses, following a current-mode approach. However, rather than using a single pFET to generate the appropriate I w current, via the translinear principle (Gilbert, 1975), it uses a differential pair in negative feedback configuration. This allows the circuit to achieve LPF functionality with tunable dynamic conductances: Input voltage pulses are integrated to produce an output current that has maximum amplitude set by V w, V t , and V thr . In all circuits of Figure 1 the V w bias (the synaptic weight) can be set by local circuits to implement learning and plasticity (Fusi et al., 2000;Mitra et al., 2009). However, the DPI offers an extra degree of freedom via the V thr bias. This parameter can be used to implement additional adaptation and plasticity schemes, such as intrinsic or homeostatic plasticity (Bartolozzi and Indiveri, 2009). A complete analysis of the DPI and its modes of operation is provided in (Bartolozzi and Indiveri, 2007). The DPI will be used in Section 4.2 to implement the DPI-neuron.

Thermodynamically equivalent models
Many of the membrane channels that shape the output activity of a neuron exhibit dynamics that can be represented by state changes of a series of voltage-dependent gating particles, which must be open for the channel to conduct. The state-transitions of these particles can be understood within the context of thermodynamic equivalent 3 SIlIcon neuron cIrcuIt blockS

Temporal integration
It has been shown that an efficient way of modeling neuron conductance dynamics and synaptic transmission mechanisms is by using simple first-order differential equations of the type t  y y x = − + , where y represents an output voltage or current, and x the input driving force (Destexhe et al., 1998). For example, this equation governs the behavior of all passive ionic channels found in nerve membranes. In the classical silicon neuron implementation proposed by Mahowald and Douglas (1991) the circuit used to implement the equation described above for modeling the neuron's passive leak conductance is the follower-integrator circuit. The follower-integrator comprises a transconductance amplifier configured in negative feedback mode with its output node connected to a capacitor. When used in the weak-inversion domain, as a voltage mode circuit, the follower-integrator behaves as a first-order low-pass filter with a tunable conductance. A detailed description of this circuit is provided in Liu et al. (2002). Conversely, in current-mode designs, an efficient strategy for implementing the first-order differential equations described above, is to use log-domain circuits (Tomazou et al., 1990). For example, the log-domain "Bernoulli-Cell" is a circuit that can implement synaptic and conductance dynamics (Drakakis et al., 1997). The circuit operates in current-mode and in the weak-inversion (or sub-threshold) domain. It has been fully characterized in Drakakis et al. (1997), and has been used to implement Hodgkin-Huxley VLSI models of neurons (Toumazou et al., 1998). A similar log-domain circuit is shown in Figure 1A: This circuit, called the "Tau-Cell," was first proposed in Edwards and Cauwenberghs (2000) as a BiCMOS log-domain filter; it was fully characterized in van Schaik and Jin (2003) as a sub-threshold log-domain circuit, and used in Yu and Cauwenberghs (2010b) to implement conductance-based synapses. This circuit is used also in the tau-cell neuron, described in Section 4.2. Another sub-threshold log-domain circuit is the low pass filter (LPF) described in Boahen (2004, 2007), and shown in Figure 1B. This circuit is based on the standard log-domain low pass filter (Frey, 1993) originally implemented using bipolar transistors, but has been simplified to act as a voltage pulse integrator: Input voltage pulses
Indiveri et al. Neuromorphic silicon neurons sub-threshold circuits described above, and integrators are typically implemented using classical filter design techniques or S-C techniques. Examples of integrators implemented using these design strategies are described in Section 4.4.

SpIke-event generatIon
Biophysically realistic implementations of neurons produce analog waveforms that are continuous and smooth in time, even for the generation of action potentials (we will describe examples of these types of circuits in Section 4.1). In many other neuron models, however, the action potential is a discontinuous and discrete event which is generated whenever a set threshold is crossed. One of the original circuits proposed for generating discrete events in VLSI implementations of silicon neurons is the Axon-Hillock circuit (Mead, 1989). Figure 3A shows a schematic diagram of this circuit. The amplifier block A is typically implemented using two inverters in series. Input currents I in are integrated on the membrane input capacitance C mem , and the analog voltage V mem increases linearly until it reaches the amplifier switching threshold (see Figure 3B). At this point V out quickly changes from 0 to V dd , switching on the reset transistor and activating a positive feedback through the capacitor divider implemented by C mem and the feedback capacitor C fb . If the reset current set by V pw is larger then the input current, the membrane capacitor is discharged, until it reaches the amplifier's switching threshold again. At this point V out swings back to 0 and the cycle repeats. The inter-spike interval t L is inversely proportional to the input current, while the pulse duration period t H depends on both the input and reset currents. A comprehensive description of the circuit operation is presented in Mead (1989).
One of the main advantages of this self-resetting neuron circuit are its excellent matching properties: mismatch is mostly dependent on the matching properties of the two capacitors of the circuit rather than any of its transistors. As low mismatch is especially desirable in imagers and photoreceptor arrays, this circuit has been applied to the design of a spiking (or event-based) vision sensor (Azadmehr et al., 2005;Olsson and Häfliger, 2008). In this case, rather than using the reset voltage V pw as an analog bias, the designers used it as a digital signal externally controlled, and exploited its good matching properties. models (Destexhe and Huguenard, 2000): The membrane voltage creates an energy barrier which a gating particle (a charged molecule) must overcome to change states (e.g., to open). Changes in the membrane voltage modulate the size of the energy barriers, altering the rates of opening and closing of a gating particle. The average conductance of a channel is proportional to the percentage of the population of individual channels that are open.
Since transistors also involve the movement of a charged particle through an electric field, a transistor circuit can directly represent the action of a population of gating particles (Hynna and Boahen, 2007). Figure 2 shows a thermodynamic model of a gating variable in which the drain current of transistor M2 in Figure 2A represents the gating particle's rate of opening, while the source current of M1 represents the rate of closing. The voltage V O controls the height of the energy barrier in M2: Increasing V O increases the opening rate, shifting u V toward u H . Increasing V C has the opposite effect: The closing rate increases, shifting u V toward u L . Generally, V O and V C are inversely related; that is, as V O increases, V C should decrease.
The source of M2, u V is the log-domain representation of the gating variable u. Attaching u V to the gate of a third transistor (not shown) realizes the variable u as a modulation of a current set by u H . Connected as a simple activating channel -with V O proportional to the membrane voltage (Hynna and Boahen, 2007) -the voltage dependence of the steady-state and time-constant of u, as measured through the output transistor, match the sigmoid and bell-shaped curves commonly measured in neurophysiology (see Figure 2B). This circuit will be used in Section 4.1 to implement the Thalamic relay neuron.

Phenomenological models
It is also possible to model conductance and channel dynamics by abstracting their behavior, describing it with sets of differential equations, and solving them using analog circuits. One can resort to using systematic synthesis methods for mapping non-linear differential equations onto analog circuits. For example, using this strategy it was possible to design circuit implementations for the FitzHugh-Nagumo neuron model (FitzHugh, 1961), as proposed in Linares- Barranco et al. (1991). These methods typically use voltage mode above-threshold circuits rather than the current-mode potassium conductance creates the downswing. In the circuit this is modeled as follows: As V mem rises above V thr , the output voltage of the comparator will rise to the positive power supply. The output of the following inverter will thus go low, thereby allowing the sodium current I Na to pull up the membrane potential. At the same time however, a second inverter will allow the capacitance C K to be charged at a speed which can be controlled by the current I Kup .
As soon as the voltage on C K is high enough to allow conduction of the nFET M2, the potassium current I K will be able to discharge the membrane capacitance. Two different potassium channel currents govern the opening and closing of the potassium channels: The current I Kup controls the spike width, as the delay between the opening of the sodium channels and the opening of the potassium channels is inversely proportional to I Kup . If V mem now drops below V thr , the output of the first inverter will become high, cutting off the current I Na . Furthermore, the second inverter will then allow C K to be discharged by the current I Kdn . If I Kdn is small, the voltage on C K will decrease only slowly, and, as long as this voltage stays high enough to allow I K to discharge the membrane, it will be impossible to stimulate the neuron for I ex values smaller than I K . Therefore I Kdn controls the refractory period of the neuron. The principles used by this design to control spiking thresholds explicitly have been used in analogous SiN implementations (Indiveri, 2000;Indiveri et al., 2001;Liu et al., 2001). Similarly, the principle of using starved inverters 1 and capacitors to implement refractory periods is used also in the DPI neuron described in Section 4.2.

SpIkIng threSholdS and refractory perIodS
The Axon-Hillock circuit produces a spike event when the membrane voltage crosses a voltage threshold that depends on the geometry of the transistors and on the VLSI process characteristics. In order to have better control over the spiking threshold, it is possible to use a five-transistor amplifier, as shown in Figure 4A. This neuron circuit, originally proposed in (van Schaik, 2001) comprises circuits for both setting explicit spiking thresholds and implementing an explicit refractory period. Figure 4B depicts the various stages that the membrane potential V mem is involved in, during the generation of an action potential.
The capacitance C mem of this circuit models the membrane of a biological neuron, while the membrane leakage current is controlled by the gate voltage V lk , of an nFET. In the absence of any input the membrane voltage will be drawn to its resting potential (ground, in this case), by this leakage current. Excitatory inputs (e.g., modeled by I in ) add charge to the membrane capacitance, whereas inhibitory inputs (not shown) remove charge from the membrane capacitance. If an excitatory current larger than the leakage current is injected, the membrane potential V mem will increase from its resting potential. The voltage V mem is compared with the threshold voltage V thr , using a basic transconductance amplifier (Liu et al., 2002). If V mem exceeds V thr , an action potential is generated. The generation of the action potential happens in a similar way as in the biological neuron, where an increased sodium conductance creates the upswing of the spike, and a delayed increase of the with the individually separated dendritic branches, allowing for parallel processing of different sets of inputs on different branches before their outputs are combined (Mel, 1994). Early VLSI dendritic systems included the passive cable circuit model of the dendrite specifically by implementing the dendritic resistance using S-C circuits (Elias and Northmore, 1999;Rasche and Douglas, 2001). Other groups have subsequently incorporated some active channels into VLSI dendritic compartments [e.g., (Arthur and Boahen, 2004)]. Farquhar and Hasler applied their transistor channel approach for building ion channels (Farquhar et al., 2004) to building active dendrite models in which ions were able to diffuse both across the membrane and axially along the length of the dendrite (Hasler et al., 2007). They used sub-threshold MOSFETs to implement the conductances seen along and across the An additional advantage that this circuit has over the Axon-Hillock circuit is power consumption: The Axon-Hillock circuit non-inverting amplifier, comprising two inverters in series, dissipates large amounts of power for slowly varying input signals, as the first inverter spends a significant amount of time in its fully conductive state (with both nFET and pFET conducting) when its input voltage V mem slowly crosses the switching threshold. The issue of power consumption has been addressed also in other SiN designs, and will be discussed in Section 4.1.

SpIke-frequency adaptatIon and adaptIve threSholdS
Spike-frequency adaptation is a mechanism observed in a wide variety of neural systems. It acts to gradually reduce the firing rate of a neuron in response to constant input stimulation. This mechanism may play an important role in neural information processing, and can be used to reduce power consumption and bandwidth usage in VLSI systems comprising networks of silicon neurons.
There are several processes that can produce spike-frequency adaptation. Here we will focus on the neuron's intrinsic mechanism which produces slow ionic currents with each action potential that are subtracted from the input. This "negative feedback mechanism" has been modeled differently in a number of SiNs.
The most direct way of implementing spike-frequency adaptation in a SiN is to integrate the spikes produced by the SiN itself (e.g., using one of the filtering strategies described in Section 3.1) and subtract the resulting current from the membrane capacitance. This would model the effect of calcium-dependent after-hyperpolarization potassium currents present in real neurons (Connors et al., 1982) and introduce a second slow variable in the model, in addition to the membrane potential variable, that could be effectively used to produce different spiking behaviors. Figure 5A shows measurements from a SiN with this mechanism implemented (Indiveri, 2007), in response to a constant input current.
Spike-frequency adaptation and other more complex spiking behaviors can also be modeled by implementing models with adaptive thresholds, as in the Mihalas-Niebur neuron model ). In this model a simple first-order equation is used to update the neuron's spiking threshold voltage based on the membrane voltage variable itself: For high membrane voltage values, the spiking threshold adapts upwards, increasing the time between spikes for a constant input. Low membrane voltage values, on the other hand, result in a decrease of the spiking threshold voltage. The speed at which the threshold adapts in this model is dependent on several parameters. Tuning of these parameters determines the type of spiking behavior that is exhibited by the SiN. Figure 5B shows spike-frequency adaptation using an adaptive threshold. Here each time the neuron spikes the threshold voltage resets to a higher value so that the membrane voltage must grow by a larger amount and hence the time between spikes increases.
Examples of two-state variable SiNs that use either of these mechanisms will be presented in Section 4.

axonS and dendrItIc treeS
Recent experimental evidence suggests that individual dendritic branches can be considered as independent computational units. A single neuron can act as a multi-layer computational network, demonstrate that the response of a dendritic component can be described as a non-linear sigmoidal function of both input temporal synchrony and spatial clustering (Wang and Liu, 2010). This response function means that linear or non-linear computation in a neuron can be evoked depending on the input spatio-temporal pattern.

Digi-MOS
Circuits that operate like a MOS transistor but with a digitally adjustable size factor W/L are very useful in neuromorphic SiN circuits, for providing a weighted current or for calibration to compensate for mismatch. Figure 7 shows a possible circuit implementation membranes and model diffusion as the macro-transport method of ion flow. The resulting single dimensional circuit is analogous to the diffuser circuit described in Hynna and Boahen (2006), but allows the conductances of each of the MOSFETs to be individually programmed to obtain the desired neuron properties. In Hasler et al. (2007) they showed how an aVLSI active dendrite model could produce action potentials down a cable of uniform diameter with active channels every five segments. The authors in Wang and Liu (2010) have recently constructed an aVLSI neuron with a reconfigurable dendritic architecture which includes both individual computational units and a different spatial filtering circuit (see Figure 6). Using this VLSI prototype, they

SIlIcon neuron ImplementatIonS
We will now make use of the circuits and techniques introduced in Section 3 to describe silicon neuron implementations. We organized the various circuit solutions in the following way: sub-threshold biophysically realistic models; compact I&Fcircuits for event-based systems; generalized I&F neuron circuits; above threshold, accelerated-time, S-C, and digital designs.

Sub-threShold bIophySIcally realIStIc modelS
The types of SiN designs described in this section exploit the biophysical equivalence between the transport of ions in biological channels and charge carriers in transistor channels. In the classical conductance-based SiN implementation described in Mahowald and Douglas (1991), the authors modeled ionic conductances using five-transistor transconductance amplifier circuits (Liu et al., 2002). In Farquhar and Hasler (2005), the authors showed how it is possible to model ionic channels using single transistors, operated in the sub-threshold domain. By using two-transistor circuits Hynna and Boahen (2007) showed how it is possible to implement complex thermodynamic models of gating variables (see also Section 3.1). By using multiple instances of the gating variable circuit of Figure 2A it is possible, for example, to build biophysically faithful models of thalamic relay neurons.

The thalamic relay neuron
Thalamic relay neurons possess a low-threshold calcium channel (also called a T-channel) and a slow inactivation variable, which turns off at higher voltages and opens at low voltages. The T-channel can be implemented using a fast activation variable, and implemented using the gating variable circuit of Figure 2. Figure 8A shows a simple two-compartment neuron circuit with a T-channel current, which can reproduce many response properties of real Thalamic relay cells (Hynna and Boahen, 2009). In the based on MOS ladder structures (Linares- . In this example, the five-bit control word b 4 b 3 b 2 b 1 b 0 is used to set the effective (W/L) eff ratio. As the currents flowing through each subbranch differ significantly, this circuit does not have unique timeconstants. Furthermore small currents flowing through the lower bit branches will settle to a steady state value very slowly, therefore such a circuit should not be switched at high-speeds, but should rather be used to provide DC biasing currents. This circuit has been used in spatial contrast retinas [18] and charge packet I&F neurons within event-based convolution chips (Serrano-Gotarredona et al., 2006 for mismatch calibration. Alternative design schemes, using the same principle but different arrangement of the transistors can be used for applications in which high-speed switching is required (Leñero-Bardallo et al., 2010).

Very low current mirrors
Typically, the smallest currents that can be processed in conventional circuits are limited by the MOS "off sub-threshold current," which is the current a MOS transistor conducts when its gateto-source voltage is zero. However, MOS devices can operate well below this limit (Linares- . To make MOS transistors operate properly below this limit, one needs to bias them with negative gate-to-source voltages, as illustrated in the current mirror circuit of Figure 7C. Transistors M1-M2 form the current mirror. Current I in is assumed to be very small (pico or femto amperes), well below the "off sub-threshold current." Consequently, transistors M1 and M2 require a negative gate-to-source voltage. By using the voltage level shifter M4-M5 and connecting the source voltage of M1-M2 to V nsh = 0.4 V, the mirror can be biased with negative gate-to-source voltages. This technique has been used to build very low frequency compact oscillators and filters (Linares- , or to perform in-pixel direct photo current manipulations in spatial contrast retinas (Costas-Santos et al., 2007). inhibitory current is input into the cell, lowering the initial membrane voltage, then the T-channel deactivates prior to the step (see Figure 8B). Once the step occurs, V mem begins to slowly increase until the T-channel activates, which excites the cell and causes it to burst. Since V mem is now much higher, the T-channel begins to inactivate, seen in the decrease of spike frequency within the burst on successive spikes, leading eventually to a cessation in spiking activity. In addition to the behavior shown here, this simple model also reproduces the Thalamic response to sinusoidal inputs (Hynna and Boahen, 2009).
The approach followed for this Thalamic relay SiN can be extended by using and combining multiple instances of the basic building blocks described in Section 3.1.

A sub-threshold Hodgkin-Huxley based neuron
In Yu and Cauwenberghs (2010a) the authors proposed a sub-threshold Hodgkin-Huxley (H-H) based SiN model by combining instances of the tau-cell circuit shown in Figure 1 with sub-threshold circuits of the type shown in Figure 9, which implement the non-linear functions typically used with the H-H model gating variables m, h, and n (Hodgkin and Huxley, 1952). Specifically, in Yu and Cauwenberghs (2010a) the authors presented a mixed-signal VLSI chip integrating a biophysical network of four H-H neurons and twelve conductancebased synapses, with programmable detailed kinetics of channel gating variables. The voltage dependence profile of closing and opening neuron circuit of Figure 8A the first block (on the left) integrates input spikes and represents the dendritic compartment, while the second block (on the right) produces output voltage spikes, and represents the somatic compartment.
The dendritic compartment contains all active membrane components not involved in spike generation -namely, the synapses (e.g., one of the low-pass filters described in Section 3.1) and the T-channel -as well as common passive membrane components -a membrane capacitance (C mem ) and a membrane conductance (the nFET M1).
The somatic compartment, comprising a simple I&F neuron such as the Axon-Hillock circuit described in Section 3.2, receives input current from the dendrites through a diode-connected transistor (M2). Though a simple representation of a cell, relay neurons respond linearly in frequency to input currents (McCormick and Feeser, 1990), just as an I&F cell. Due to the rectifying behavior of the diode (the pFET M2 in Figure 8A), current only passes from the dendrite to the soma. As a result, the somatic action potential does not propagate back to the dendrite; only the hyperpolarization (reset) that follows is evident in the dendritic voltage trace (V mem ). This is a simple approximation of dendritic low-pass filtering of the back-propagating signal.
When V mem rests at higher voltages, the T-channel remains inactivated, and a step change in the input current simply causes the cell to respond with a constant frequency (see Figure 8C). If an  rent between V dd and ground (GND) to flow when the transistors are both on and in saturation, which is the case at threshold. This is further compounded by the fact that the membrane potential usually changes very slowly, on time scales of milliseconds to seconds, which means that the spike generator remains in the high power consumption regime. Positive feedback, either capacitive-or current-based, can accelerate its transition. Capacitive feedback is already used in the Axon-Hillock circuit, however current-based feedback is more effective for reducing power consumption. Hence the octopus neuron has four interesting properties: (1) It uses current feedback to accelerate the membrane potential transition when threshold is reached by adding an additional input current to the neuron.
(2) It efficiently re-uses the short-circuit current in the spike generator to generate the feedback current. (3) The membrane capacitor (C mem of Figure 10) is not completely discharged to GND during spike production by disconnecting it from the feedback current and the input of the spike generator. The input of the spike generator, however, accelerates to G nd . This reduces power during spike production by a factor of approximately 25, accelerates the transition at threshold by a factor of approximately 100, and reduces power consumption during reset (Culurciello et al., 2003). (4) It only consumes power during spike generation and reset, which typically lasts for a few nano-seconds. The net effect is a total energy consumption of less than 4 pJ/spike in the 0.6 μm CMOS process in which the chip was implemented.

The dynamic vision sensor differencing neuron
Another compact neuron circuit is the one used in the dynamic vision sensor (DVS) silicon retina (Lichtsteiner et al., 2008). This circuit is optimized to reduce mismatch across cells. The DVS has pixels that produce an ON or OFF event signifying quantized increases and decreases of log intensity since the last event from the pixel. In the DVS, the input to the ON/OFF neuron comes from a logarithmic photoreceptor, but this same circuit could be used with any input that can drives the capacitive input. This circuit is nearly a perfect rates for each of the 24 channel gating variables are individually digitally programmable using on-chip digital-to-analog converters (DACs) and analog spline regression functions implemented with the seven-point additive spline (Yu and Cauwenberghs, 2010a) sigmoidal function circuit of Figure 9. Tau-cell based dynamic and cascaded translinear circuits ( Figure 1A) implement first-order rate kinetics in the channel variables and their non-linear gating of the corresponding membrane channel conductances. The comparison between the experimental silicon and modeled (experimental neuroscience) equilibrium values of the channel gating variables is given in Figure 9B. The temporal scale of the dynamics both in the membrane and the channel variables can be uniformly scaled, for a global speedup of the analog simulation, by tuning a single current bias parameter (Yu and Cauwenberghs, 2010a).

compact Integrate-and-fIre cIrcuItS for event-baSed SyStemS
We have shown examples of circuits used to implement faithful models of spiking neurons. These circuits can require significant amounts of silicon real-estate. At the other end of the spectrum are compact circuits that implement basic models of I&F neurons. A common goal is to integrate very large numbers of these circuits on single chips to create large arrays of spiking elements, or large networks of neurons densely interconnected (Merolla et al., 2007;Vogelstein et al., 2007;Schemmel et al., 2008). In these systems, the strategy used to transmit spikes off-chip is to use the addressevent representation (AER; Lazzaro et al., 1993;Deiss et al., 1998;Boahen, 2000): Each spiking neuron is assigned an address and when a neuron fires its address is instantaneously put on a digital bus, using asynchronous digital circuits that map and route the spikes to other nodes on different chips (Chicca et al., 2007;Schemmel et al., 2008). In this representation time represents itself, and analog signals are encoded by the inter-spike intervals between the addresses of their sending neurons. It is therefore important to develop compact low-power circuits that implement useful abstractions of real neurons, but that can also produce very fast digital pulses required by the asynchronous circuits that manage the AER communication infrastructure.
A common application of basic I&F spiking circuits is their use in neuromorphic vision sensors. In this case the neuron is responsible for encoding the signal measured by the photoreceptor, and transmitting it off-chip using the AER. In Azadmehr et al. (2005) and Olsson and Häfliger (2008), the authors used the Axon-Hillock circuit described in Section 3.2 to produce AER events. In Olsson and Häfliger (2008) the authors showed how this circuit can be interfaced to the AER interfacing circuits in a way to minimize device mismatch. Conversely, in Culurciello et al. (2003) the authors developed an imager inspired by the octopus retina, in which the spiking neuron circuit was optimized for minimum power consumption. In Lichtsteiner et al. (2008), the authors developed a retina for sensing changes in brightness, using a compact ON/OFF neuron with good threshold matching properties.

The octopus retina neuron
The neuron used in the octopus retina (Culurciello et al., 2003) is shown in Figure 10. As mentioned in Section 3.2, any neuron that uses inverters (starved or otherwise) will allow the short-circuit cur- FIgure 10 | The octopus retina neuron. The input current is generated by a photodetector, while the spike generator uses positive current feedback to accelerate input and output transitions to minimize short-circuit currents during spike production. The membrane capacitance (C mem ) is disconnected from the input of the spike generator to further accelerate transition and to reduce power during reset. approaches can be obtained by implementing conductance-based or generalized I&F models (Jolivet et al., 2004). It has been shown that these types of models that capture many of the properties of biological neurons, but require less and simpler differential equations compared to H-H based models (Izhikevich, 2003;Jolivet et al., 2004;Brette and Gerstner, 2005;Mihalas and Niebur, 2009;Naud et al., 2009). In addition to being efficient computational models for software implementations, these models lend themselves to efficient hardware implementation as well (Wijekoon and Dudek, 2008;Folowosele et al., 2009a;Livi and Indiveri, 2009;Indiveri et al., 2010;Rangan et al., 2010;van Schaik et al., 2010avan Schaik et al., , 2010b.

The tau-cell neuron
The circuit shown in Figure 12, dubbed as the "Tau-Cell neuron" been used as the building block for implementations of both the Mihalas-Niebur neuron (van Schaik et al., 2010a) and the Izhikevich neuron (Rangan et al., 2010;van Schaik et al., 2010b). The basic leaky I&F functionality is implemented using the taucell log-domain circuit described in Section 3.1. This approach uses current-mode circuits, so the state variable, which is normally the membrane voltage, V mem , is transformed to a current I mem . A tau-cell, configured as a first-order low-pass filter, is used to model the leaky integration. In order to create a spike, I mem is copied by pFETs M5 and M8 and compared with the constant threshold current I u . Since I mem can be arbitrarily close to I u , a current limited inverter (M12, M13) is added to reduce power consumption while converting the result of the comparison into a digital value V nspk . A positive voltage spike V spk is generated with inverter M14, M15 with a slight delay with respect to V nspk . pFET M5-M7 implement positive feedback based on V nspk while nFET M16 resets I mem to a value determined by V el . This reset causes the end of the positive feedback and the end of the spike and the membrane is ready to start the next integration cycle.

The log-domain LPF neuron
The log-domain LPF neuron (LLN) is a simple yet reconfigurable I&F circuit Boahen, 2004, 2007) that can reproduce many of the behaviors expressed by generalized I&F models. Based integrator: The corner frequency is about 0.05 Hz and is limited by the off-state leakage of the reset transistor. The pixel circuit is shown in Figure 11. Each time the pixel outputs an event (either ON or OFF), a reset pulse from the AER communication circuits memorizes the last log intensity value across capacitor C 1 . Changes in log intensity are capacitively coupled to the input of the inverting capacitive-feedback amplifier A 1 , which has a gain of about −20. The A 1 output V d1 is then compared to two reference levels by the high gain amplifiers A ON and A OFF . When V ON or V OFF crosses the logic threshold, transmission of the ON or OFF event is initiated, resulting finally in a pulse on V reset that starts the cycle over again.
The matching behavior of this circuit is the key to the success of the DVS, which is the first event-based silicon retina that has been commercialized and sold to other institutions. Because the DC mismatch in the log intensity value is blocked by C 1 , and because A 1 inserts a high gain element that appears before the poorly matched comparators A ON and A OFF , the mismatch referred back to the signal of interest (dlogI) is reduced by the gain of A 1 . For example, if the mismatch of A ON /A OFF are 20 mV and the gain A 1 = 20, then the mismatch at the logI output is reduced to 1 mV, which corresponds to a visual contrast of about 3.5%. This relatively good matching allows the DVS to be used with natural visual input, which often has rather low contrast. This circuit is an example of the general principle of removing static mismatch and amplifying before comparing for improving precision using imprecise elements. Measurements show that across an array of 16k pixels the one-sigma matching is equivalent to about 2% contrast. The five-sigma matching (which applies across a large array of cells) is then about 10%, in agreement with practical contrast threshold settings of about 15% that we routinely use (Lichtsteiner et al., 2008).

generalIzed Integrate-and-fIre neuron cIrcuItS
The simplified I&F neuron circuits described in the previous Section require far less transistors and parameters than the biophysically realistic models of Section 4.1. But they do not produce a rich enough repertoire of behaviors useful for investigating the computational properties of large neural networks (Izhikevich, 2003;Brette and Gerstner, 2005 logI FIgure 11 | Dynamic vision sensor (DVS) complementary oN-oFF differencing neuron. ON and OFF events are produced by quantized increases and decreases of log intensity since the last event from the pixel. The ON/OFF events are reset by row and column acknowledge signals V CA and V RA , which produces a reset pulse of a duration controlled by V refr . The mismatches in the comparator thresholds of ≈20 mV is reduced by a factor of the gain of 20 in the switched-capacitor differencing amplifier, resulting in an effective threshold mismatch of about 1 mV at Vp. dimensionless membrane potential v, and adaptive conductance g variable (proportional to I n and I g of Figure 13A respectively), can be described by the following set of equations: where n ∞ is v's steady state level in the absence of positive feedback and g = 0; t and t g are the membrane and adaptive conductance time-constants, respectively; and g max is the adaptive conductance's absolute maximum value. When v reaches a high level (>>10), a spike is emitted, and r(t) is set high for a brief period, T R . r(t) is a reset-refractory signal, driving v low (not shown in equation). The LLN is composed of four sub-circuits (see Figure 13): A membrane LPF (M L1-3 ), a spike event generation and positivefeedback element (M A1-6 ), a reset-refractory pulse generator (M R1-3 ), and an adaptation LPF (M G1-4   on the LPF of Figure 1B, the LLN benefits from the log-domain design style's efficiency, using few transistors, operating with lowpower (50-1000 nW), and requiring no complex configuration. The LLN realizes a variety of spiking behaviors: Regular spiking, spike-frequency adaptation, and bursting ( Figure 13B). The LLN's where I mem is the sub-threshold current analogous to the state variable v of Eq. (1) and I g corresponds to the slow variable g of Eq.
(1) responsible for spike-frequency adaptation. The term f(I mem ) accounts for the positive-feedback current I a of Figure 14 and is an exponential function of I mem (Indiveri et al., 2010; see also Figure 14B). As for the LLN, the function r(t), is unity for the period in which the neuron spikes, and null in other periods. The other parameters in Eq. 2 are defined as: By changing the biases that control the neuron's time-constants, refractory period, spike-frequency adaptation dynamics and leak behavior (Indiveri et al., 2010) the DPI-neuron can produce a wide range of spiking behaviors ranging from regular spiking to bursting.
Indeed, given the exponential nature of the generalized I&F neuron's non-linear term f (I mem ), the DPI-neuron implements an adaptive exponential I&F model (Brette and Gerstner, 2005). This I&F model has been shown to be able to reproduce a wide range of spiking behaviors, and explain a wide set of experimental measurements from pyramidal neurons (Brette and Gerstner, 2005). For comparison the LLN uses a cubic term, while the tau-cell based neuron circuits proposed in (Rangan et al., 2010;van Schaik et al., 2010b) and the quadratic and the S-C SiNs described in Section 4.4 use a quadratic term (implementing the I&F computational models proposed by Izhikevich, 2003).

above threShold, accelerated-tIme, SwItched-capacItor, and dIgItal deSIgnS
The SiN circuits described up to now have transistors that operate mostly in the sub-threshold or weak-inversion domain, with currents ranging typically between fractions of pico to hundreds of nano-amperes. These circuits have the advantage of being able to emulate real neurons with extremely low power requirements and with realistic time-constants (e.g., for interacting with the nervous system, or implementing real-time behaving systems with timeconstants matched to those of the signals they process). However, in the weak-inversion domain mismatch effects are more pronounced than in the strong-inversion regime (Pelgrom et al., 1989), and often require learning, adaptation or other compensation schemes.
It has been argued that in order to faithfully reproduce computational models simulated on digital architectures, it is necessary to design analog circuits with low mismatch and high precision (Schemmel et al., 2007). For this reason, several SiN circuits that operate in the strong-inversion regime have been proposed. In this regime however, currents are four to five orders of magnitude larger. With such currents, the active circuits used to implement resistors decrease their resistance values dramatically. As passive resistors cannot be easily implemented in VLSI to yield large resistance values, it is either necessary to use large off-chip capacitors (and small numbers of neurons per chip), to obtain biologically realistic time-constants, or to use "accelerated" time scales, in (∝n ∝ ). The positive-feedback element drives the membrane LPF in proportion to the cube of v, analogous to a biological sodium channel population. When the membrane LPF is sufficiently driven, n n 3 3 > , resulting in a run-away potential, i.e., a spike. The digital representation of the spike is transmitted as an AER request (REQ) signal. After a spike (upon arrival of the AER acknowledge signal ACK), the refractory pulse generator creates a pulse, r(t) with a tunable duration. When active r(t) turns M G1 and M R3 ON, resetting the membrane LPF (toward V DD ) and activating the adaptation LPF. Once activated the adaptation LPF inhibits the membrane LPF, realizing I g (∝ g), which is proportional to spike frequency.
Implementing LLN's various spiking behaviors is a matter of setting its biases. To implement regular spiking, we set g max = 0 (set by M G2 's bias voltage V wahp ) and T R = 1 ms (long enough to drive v to 0, set by M R2 's bias voltage V ref ). Spike-frequency adaptation can be obtained by allowing the adaptation LPF (M G1-4 ) to integrate the spikes produced by the neuron itself. This is done by increasing g max and setting t g = 100 ms (i.e., by adjusting V lkahp appropriately). Similarly, the bursting behavior is obtained by decreasing the duration of the r(t) pulse such that v is not pulled below 1 after each spike.

The DPI neuron
The DPI neuron is another variant of a generalized I&F model (Jolivet et al., 2004). This circuit has the same functional blocks used by LLN of Figure 13, but different instantiations of low-pass filters and current-based positive-feedback circuits: The low-pass filter behavior is implemented using instances of the tunable Diff-Pair Integrator circuit described in Section 3.1, while the positive feedback is implemented using the same circuits used in the Octopus Neuron of Figure 10. These are small differences from the point of view of transistor count and circuit details, but have an important effect on the properties of the SiN.
The DPI-neuron circuit is shown in Figure 14A. It comprises an input DPI filter (M L1 − M L3 ), a spike event generating amplifier with current-based positive feedback (M A1 − M A6 ), a spike reset circuit with AER handshaking signals and refractory period functionality (M R1 − M R6 ), and a spike-frequency adaptations mechanism implemented by an additional DPI filter (M G1 − M G6 ). The input DPI filter M L1 − M L3 models the neuron's leak conductance, producing exponential sub-threshold dynamics in response to constant input currents. The integrating capacitor C mem represents the neuron's membrane capacitance, and the positive-feedback circuits in the spike-generation amplifier model both sodium channel activation and inactivation dynamics. The reset and refractory period circuit models the potassium conductance functionality. The spike-frequency adaptation DPI circuit models the neuron's calcium conductance, and produces an after hyperpolarizing current (I g ) proportional to the neuron's mean firing rate.
By applying a current-mode analysis to both the input and the spike-frequency adaptation DPI circuits (Bartolozzi et al., 2006;Livi and Indiveri, 2009), it is possible to derive a simplified analytical solution (Indiveri et al., 2010), very similar to the one described in Eq. (1), of the form: circuits that implement the exact equations of Hodgkin-Huxley conductance models offer the opportunity to select and tune precisely each of the parameters appearing in the set of non-linear differential equations. These circuits need to model conductances for channels of the following types: voltage-gated, voltage and concentration-gated, passive leak, and synaptic voltage-gated. It is possible to reproduce a large variety of neural activity patterns by using a small set of bipolar and above threshold analog circuits and adjusting their parameters to represent different biophysical properties. Such types of circuits have been designed following a current-mode approach, fabricated using SiGe 0.8 and 0.35 μm technologies, and fully characterized in Renaud et al. (2007) and Saighi et al. (2011).
which the time-constants of the SiNs are a factor of 10 3 or 10 4 smaller than those of real neurons. Alternatively, one can use S-C for implementing small conductances (and therefore long timeconstants) by moving charge in and out of integrated capacitors with clocked switches. Taking this concept one step further, one can implement SiNs using full-custom clocked digital circuits. All of these approaches are outlined in this section.

Above threshold Hodgkin-Huxley models
As mentioned in Section 4.1, Hodgkin-Huxley (H-H) conductance-based models describe faithfully the biophysics of excitable cells and are helpful to capture the main intrinsic firing and response properties of neurons. Above-threshold and bipolar Response of the DPI neuron circuit to a constant input current. The measured data was fitted with a function comprising an exponential ∝e −t/tK at the onset of the stimulation, characteristic of all conductance-based models, and an additional exponential ∝e +t/tNa (characteristic of exponential I&F computational models; Brette and Gerstner, 2005) at the onset of the spike (Indiveri et al., 2010). costly in terms of silicon area and time-to-fabrication, due to the full-custom design mode and to the open parameter space that necessitates above-threshold design with bipolar and MOS transistors. To improve the design flow, the analog circuits are designed as library items which form a database. The database is used as a platform for automated design (Lévi et al., 2008): an automatic exploration process searches the database and helps a designer reuse library circuits for new designs, according to the specifications provided (e.g., from a conductance equation).
Neuron membrane voltages are obtained by summing currents chosen from a set of "generator" library circuits and summed on a capacitance representing a membrane capacitance. The currents are selected by a system of configurable switches, and a maximum of five generators can be selected for a single neuron. This covers most of the point neuron models used in computational neuroscience. External inputs and synaptic currents from pre-synaptic neurons can be injected on the membrane capacitance. The results presented in Figure 15B were measured on an VLSI chip designed using the AMS 0.35 μm SiGe technology, simulating a four-conductance model (sodium, potassium, calcium, and leak;Saighi et al., 2011). The time-constants of the activation and inactivation variables in the sodium, potassium and calcium current-models were approximated by constants. These simplifications have only minor consequences on the model's behavior (Zou et al., 2006), as they essentially change only the shape of the spikes. These types of devices are unique tools for experiments on hybrid living-artificial neural networks. Silicon neurons represent the artificial part, connected via artificial synapses to intra-or extra-cellular electrodes to a living neural network (in vitro acute preparation or cultures). The living-artificial system acts as a single network, useful to explore cellular or synaptic mechanisms. In these types of experiments (Le Masson et al., 2002), real-time processing is mandatory to ensure a correct dialog between living and artificial neurons, and analog integrated-circuit computation simplifies the communication between the circuits and the living neurons, as the electrodes measure analog signals and inject analog currents on the living cells.

The quadratic integrate-and-fire neuron
As for the sub-threshold case, implementations of biophysically detailed models such as the one described above can be complemented by more compact implementations of simplified I&F models.
The quadratic I&F neuron circuit (Wijekoon and Dudek, 2008), shown in Figure 16A, is an example of an above-threshold generalized I&F circuit. It was inspired by the adapting quadratic I&F neuron model proposed by Izhikevich (2003). The required nonlinear oscillatory behavior is achieved using differential equations of two-state variables and a separate after-spike reset mechanism, as explained in Izhikevich (2003). However, the circuit implementation does not aim to accurately replicate the non-linear equations described in Izhikevich (2003). Instead it aims at using the simplest possible circuitry of the analog VLSI implementation that can reproduce the functional behavior of the coupled system of non-linear equations.
The two-state variables, "membrane potential" (V) and "slow variable" (U), are represented by voltages across capacitors C v and C u respectively. The membrane potential circuit consists of transistors Figure 15A shows the schematic of the analog library circuit implementing a two-parameter sigmoidal function used to implement conductance models. As the kinetics in the circuit equations are identical to the H-H model ones, ionic currents have dynamics with biologically realistic time-constants.
Each mathematical function used in the H-H neuron model, implemented using its analog equivalent circuit, is controlled by tunable analog variables which correspond to the model parameters. All parameters are stored on-chip on dynamically reconfigurable and analog DRAM cells. This implementation approach is  (c) Stimulation current I S (t). Sodium and potassium and leak conductances generate the action potentials; a calcium conductance with a slow kinetic modulates the action potential occurrence. Individual ionic currents are available for monitoring. Voltage and current scales are the biological model scales. Hardware and biological time scales are identical, as the simulation runs in continuous and real time. When the stimulation current is applied, the neuron starts oscillating and the calcium current increases, which in turn raises the oscillation frequency. At the end of the stimulation pulse, oscillations continue until the calcium current is low enough. Finally, the neuronal activity ceases.

An accelerated current-controlled conductance neuron
A design that is in-between the detailed H-H neuron circuits and the quadratic neuron circuit, in terms of transistor count and circuit complexity is the above-threshold current-controlled conductance neuron.
This circuit is also an accelerated neuron model, which uses transistors operated in the strong-inversion regime to emulate the properties of neuron membrane conductances. Together with on-chip bias-generation circuits such a model can be calibrated to quantitatively reproduce numerical simulations. Figure 17A shows an exemplary neuron circuit which is part of a 100k synapse network chip (Schemmel et al., 2006). The neuron emulates three ion-channels and a spike-generation circuit consisting of a high-speed comparator using positive feedback and an adjustable refractory period.
Functionally the ion-channels are realized by current-controlled conductances. The inhibitory and excitatory channels receive a current-sum representing the total neuro-transmitter density in the synaptic cleft of the inhibitory and excitatory synapses respectively. Thereby, the time course of the synaptic conductance is generated outside of the neuron circuit and may differ for each synapse. Using a current-mode input is mandatory at the high acceleration factor of the neuron (10 4 -10 5 ). A rise-time of 1 ms in biology translates to 10 ns. Considering a voltage swing of 1 V and a total capacitance of 5 pF for the neuron input 2 the current needed is 500 μA. If the voltage swing can be limited to 20 mV, the maximum current generated by a single synapse would be reduced to a much more manageable 10 μA.
The low input impedance necessary at the neuron inputs is generated by wide cascode transistors (M 6 and M 9 ). The circuits for the leakage and the inhibitory conductances are standard operational transconductance amplifiers (OTA1 and 2). In case of the inhibitory conductance, the linear input range is extended by using a voltage-divider chain at the input of the OTA built from long transistors. This is feasible since the additional leakage generated by these transistors can be compensated by reducing the static leakage current I leak .
The excitatory conductance has to react very quickly to changes in the input current, as shown in Figure 17B. For the post-synaptic pulse of a single synapse a current in the order of 10 μA must be sourced with a rise-time below 10 ns. Using an OTA would exceed the available silicon area and quiescent power envelope. A simple but effective solution is the usage of a current mirror (M 7 and M 8 ) with low output impedance, realized by utilizing transistors with minimum channel length. A comparison between the measured neuromorphic circuit response and a numerical simulation is shown in Figure 17B. As shown, the spike-times are in good agreement with each other. The network is in a high-conductance state throughout the stimulation.

The switched-capacitor Mihalas-Niebur neuron
Switched-capacitors have long been used in integrated circuit design to enable the implementation of variable resistors whose sizes can vary over several orders of magnitude. This technique can M1-M5 and membrane capacitor C v . The membrane capacitor integrates post-synaptic input currents the spike-generating positive-feedback current of M3, and the leakage current generated by M4 (mostly controlled by the slow variable U). The positivefeedback current is generated by M1 and mirrored by M2-M3 and depends approximately quadratically on the membrane potential. If a spike is generated, it is detected by the comparator circuit (M9-M14), which provides a reset pulse on the gate of M5 that rapidly hyperpolarizes the membrane potential to a value determined by the voltage at node c. The slow variable circuit is built using transistors M1, M2, and M6-M8. The magnitude of the current provided by M7 is determined by the membrane potential, in a way similar to the membrane circuit. The transistor M6 provides a non-linear leakage current. The transistors and capacitances are scaled so that the potential U will vary more slowly than V. Following a membrane potential spike, the comparator generates a brief pulse to turn on transistor M8 so that an extra amount of charge, controlled by the voltage at node d, is transferred onto C u . The circuit has been designed and fabricated in a 0.35-μm CMOS technology. It is integrated in a chip containing 202 neurons with various circuit parameters (transistor sizes and capacitances). As the transistors in this circuit operate mostly in strong inversion, the firing patterns are on an "accelerated" time scale, about 10 4 faster than biological real time (see Figure 16B). The power consumption of the circuit is below 10 pJ/spike. A similar circuit, but operating in weak inversion and providing spike timings on a biological time scale, has been presented in Wijekoon and Dudek (2009 be used as a method of implementing resistors in silicon neurons, which is complementary to the methods described in the previous sections. More generally, S-C implementations of SiNs produce circuits whose behaviors are robust, predictable and reproducible (properties that are not always met with sub-threshold SiN implementations). The circuit shown in Figure 18A implements a leaky I&F neuron implemented with S-Cs (Folowosele et al., 2009a). Here the post-synaptic current is input onto the neuron membrane, V m . The S-C, SW 1 , acts as the "leak" between the membrane potential, V m , and the resting potential of the neuron, E L . The value of the leak is varied by changing either the capacitor in SW 1 or the frequency of the clocks w1 and w2. A comparator (not shown) is used to compare the membrane voltage V m with a reset voltage Θ r . Once V m exceeds Θ r a "spike" voltage pulse is issued and V m is reset to the resting potential E L .
The Mihalas-Niebur S-C neuron ) is built by combining the I&F circuit of Figure 18A with the variable threshold circuit shown in Figure 18B. The circuit blocks are arranged in a way to implement the adaptive threshold mechanism described in Section 3.4. As the circuits used for realizing the membrane and the threshold equations are identical, the density of arrays of these neurons can be doubled, when simple I&F with fixed threshold properties are desired. The main drawback of this approach is the need for multiple phases of the S-C clocks which must be distributed (typically in parallel) to each neuron.
Experimental results measured from a fabricated integrated circuit implementing this neuron model (Folowosele et al., 2009b) are shown in Figure 19. The ease with which these complex behaviors can be evoked in S-C neurons, without extensive and precise tuning, demonstrates their utility in large silicon neuron arrays.

The digitally modulated charge packet input neuron
The S-C principle of using discrete time and clocked signals can be extended to use high-speed pulsing current mirrors for building weight-modulated charge packet driven leaky I&F neurons.
In this framework, spikes produced by a source neuron act as asynchronous clock signals that selectively activate a set of binary weighted high-speed pulsing current mirrors at the destination neurons. The selection of which current mirror branch is activated depends on a digital word that represents the neuron's synaptic weight. To implement the neuron leak conductance, an opposite sign pulsing current mirror is driven by spikes generated by a periodic signal from a global on chip clock. Figure 20A shows a circuit diagram representing this concept. Spikes may have a duration of down to about 100 ns for currents in the order of one nano FIgure 17 | accelerated current-controlled conductance neuron. (a) Schematic diagram: excitatory and inhibitory synaptic inputs can be connected as an array of current-sinks to the I inhib or I exc nodes. The passive leak behavior is controlled via the I leak node. (b) Measured response of the membrane potential to 256 Poisson distributed input spike trains, compared to an equivalent software simulation. The chip is calibrated to an acceleration factor of 10 4 . Top: input spike trains with 8 Hz mean firing rate in biological time. Middle: membrane voltage calculated with the software simulator NEST (Eppler et al., 2008)  amperes (Serrano-Gotarredona et al., 2006). This idea has been exploited to build arrays of I&F neurons in AER convolution chips (Serrano-Gotarredona et al., 2006). Each high-speed current mirror input branch is biased by a digitally calibrated current, using a digi-MOS (see Section 3.6), to compensate for transistor mismatch. Also, since the I&F neurons have to handle signed charge packages, both NMOS and PMOS high speed digitally calibrated pulsing current mirrors are required (see top and bottom part of Figure 20A). The neuron produces both positive (Pulse + ) and negative (Pulse − ) spikes, depending on the (excitatory or inhibitory) destination synapse, and integrates both positive and negative input pulses until it reaches corresponding positive and negative spiking thresholds. After generating an output spike, the neuron is reset to an intermediate reset level between the two thresholds. The leakage is also signed, depending on whether the actual neuron state is above or below the reset level. As both NMOS and PMOS calibrated pulsing current mirrors are available, there is no need to implement extra mirrors for the leak, but simply activate a special leak-weight when a leak-pulse (PulseF) is received. Figure 20 shows the overall neuron circuit diagram. These techniques were used in an AER convolution chip (Serrano-Gotarredona et al., 2006), for achieving an overall precision of 3 bits (plus sign). With weights and spike durations down to about 100 ns, each mirror bit branch required 5 bit calibration using transistor dimensions of 1.2/4 μm. Overall neuron size was 90 × 90 μm 2 in a 0.35 μm CMOS process.

Full-custom digital I&F neuron
An alternative option requiring approximately the same area usage, but with much higher precision, is to implement the I&F neuron using all digital techniques. This idea was explored in Camuñas Mesa et al. (2008) where the authors proposed the circuit shown in Figure 21. In this implementation a digital adder and accumulator A B FIgure 20 | (a) Digitally weight-modulated and calibrated charge packet driven leaky I&F neuron schematic. The neuron handles positive and negative charge packets to emulate excitatory and inhibitory synapses, and has input for a periodic global signal PulseF to implement a programmable constant rate leak. Neuron can deliver positive or negative output events. (b) Detail of logic block in (a).
are used together with digital comparing circuits, for implementing the integration and spike-generation operations of the I&F model. Using the same technology used for the digitally modulated neuron described above, and for a similar area usage of 100 × 100 μm 2 , it was possible to use accumulators and adders with 18-bit precision and synaptic weights with 5-bit (plus sign) precision. The I&F leak was implemented by stimulating an inhibitory synapse with a fixed weight at a periodic global rate. In this implementation spikes can have a duration as small as 50 nano-seconds.

Digital VLSI I&F neurons
Digital VLSI implementations of neurons and neural systems are also being evaluated, without resorting to full-custom VLSI designs. Examples include solutions using FPGAs (Mak et al., 2006;Cassidy and Andreou, 2008), multi-core based architectures using multiple ARM cores (Jin et al., 2010), and conventional graphical processing units (GPUs; Fidjeland et al., 2009). These approaches allow the development of large-scale spiking neural network simulations, without having to resort to powerful and power-hungry general purpose computing architectures.

dIScuSSIon
While the digital processing paradigm, ranging from standard computer simulations to custom FPGA designs, is advantageous for its stability, fast development times, and high precision properties, fullcustom VLSI solutions can often be optimized in terms of power consumption, silicon area usage, and speed/bandwidth usage. We anticipate that future developments in large-scale neuromorphic circuits and systems designs will increasingly combine full-custom analog and synthesized digital designs, in order to optimize both core and peripheral neural and synaptic functions in a highly programmable and reconfigurable architecture. The relative merits and the right mix of analog versus digital in neuromorphic computing (Sarpeshkar, 1998) remain a subject for further investigation and will likely require highly application dependent optimization. We expect such carefully tailored combinations of silicon neurons and custom analog/digital VLSI neural networks to offer solutions to a large variety of applications, ranging from the efficient implementation of large-scale and real-time spike-based computing systems, to the implementation of compact microelectronic brain-machine interfaces. In particular, even though sub-threshold current-mode circuits are reputed to have higher mismatch than above-threshold circuits, they have lower noise energy (noise power times bandwidth), and superior energy efficiency (bandwidth over power; Sarpeshkar et al., 1993;Shi, 2009). Indeed, the sources of inhomogeneities (e.g., device mismatch) which are often considered a problem, can actually be exploited in networks of SiNs for computational purposes (similar to how real neural systems exploit noise; Chicca and Fusi, 2001;Chicca et al., 2003;Merolla and Boahen, 2004). Otherwise, sources of mismatch can be minimized at the device level with clever VLSI layout techniques (Liu et al., 2002), and at the system level by using the same strategies used by the nervous system. In particular, adaptation and learning at multiple spatial and temporal scales are important mechanisms to compensate for variability in the environment, as well as in the neural hardware operating on the environment, which includes mismatch and other sources of analog imprecision in the implementation (Cauwenberghs and Bayoumi, 1999). Furthermore, by combining the advantages of synchronous and asynchronous digital technology with those of analog circuits, it is possible to efficiently calibrate component parameters and (re)configure SiN network topologies both for single chip solutions, and for large-scale multi-chip networks (Linares- Silver et al., 2007;Basu et al., 2010;Yu and Cauwenberghs, 2010a;Sheik et al., 2011).
In this paper we described some of the most common circuits and techniques used to implement silicon neurons, and described a wide range of neuron circuits that have been developed over the years, using different design methodologies and for many different application scenarios. In particular, we described circuits to implement leaky I&F neurons (Mead, 1989), adaptive threshold neurons , quadratic (Izhikevich, 2003), and adaptive exponential (Brette and Gerstner, 2005) I&F neurons, as well as conductance-based and Hodgkin-Huxley models. Table 2 lists all the SiNs described, pointing out their main features and characteristics.
Obviously, there is no absolute optimal design. As there is a wide range of neuron types in biology, there is a wide range of design and circuit choices for SiNs. While the implementations of conductance-based models can be useful for applications in which a small numbers of SiNs are required (as in hybrid systems where real neurons are interfaced to silicon ones), the compact AER I&F neurons and log-domain implementations (such as the quadratic Mihalas-Niebur neurons, the tau-cell neuron, the LPF neuron, or the DPI neuron) can be integrated with event-based communication fabric and synaptic arrays for very large-scale reconfigurable networks. Indeed, both the sub-threshold implementations and their above-threshold "accelerated-time" counterpart are very amenable for dense and low power integration with energy efficiencies of the order of a few pico-Joules per spike (Wijekoon and Dudek, 2008;Livi and Indiveri, 2009;Rangan et al., 2010). In addition to continuous time, non-clocked sub-threshold and above-thresh- old design techniques, we showed how to implement SiN using digitally modulated charge packet and S-C methodologies. The S-C Mihalas-Niebur SiN circuits is a particularly robust design which exhibits the model's generalized linear I&F properties and can produce up to ten different spiking behaviors. The specific choice of design style and SiN circuit to use depends on its application. Larger and highly configurable designs that can produce a wide range of behaviors are more amenable to research projects in which scientists explore the parameter space and compare the VLSI device behavior with that of its biological counterpart. Conversely, the more compact designs will be used in specific applications where signals need to be encoded as sequences of spikes, and where size and power budgets are critical. The sheer volume of silicon neuron designs proposed in the literature demonstrates the enormous opportunities for innovation when inspiration is taken from biological neural systems. The potential applications span computing and biology: neuromorphic systems are providing the clues for the next generation of asynchronous, low-power, parallel computing that could breach the gap in computing power when Moore's law runs its course, while hybrid, silicon-neuron systems are allowing neuro-scientists to unlock the secrets of neural circuits, leading one day, to fully integrated brain-machine interfaces. New emerging technologies (e.g., memristive devices) and their utility in enhancing spiking silicon neural networks must also be evaluated, as well as maintaining a knowledge-base of the existing technologies that have been proven to be successful in silicon neuron design. Furthermore, as larger on-chip spiking silicon neural networks are developed questions of communications protocols (e.g., AER), on-chip memory, size, programmability, adaptability, and fault tolerance also become very important. In this respect, the SiN circuits and design methodologies described in this paper provide the building blocks that will pave the way for these extraordinary breakthroughs.