Leaky Integrate-and-Fire Neuron Circuit Based on Floating-Gate Integrator

The artificial spiking neural network (SNN) is promising and has been brought to the notice of the theoretical neuroscience and neuromorphic engineering research communities. In this light, we propose a new type of artificial spiking neuron based on leaky integrate-and-fire (LIF) behavior. A distinctive feature of the proposed FG-LIF neuron is the use of a floating-gate (FG) integrator rather than a capacitor-based one. The relaxation time of the charge on the FG relies mainly on the tunnel barrier profile, e.g., barrier height and thickness (rather than the area). This opens up the possibility of large-scale integration of neurons. The circuit simulation results offered biologically plausible spiking activity (<100 Hz) with a capacitor of merely 6 fF, which is hosted in an FG metal-oxide-semiconductor field-effect transistor. The FG-LIF neuron also has the advantage of low operation power (<30 pW/spike). Finally, the proposed circuit was subject to possible types of noise, e.g., thermal noise and burst noise. The simulation results indicated remarkable distributional features of interspike intervals that are fitted to Gamma distribution functions, similar to biological neurons in the neocortex.


INTRODUCTION
Ongoing research efforts into spiking neural networks (SNNs) attempt to gain a better understanding of the brain (Gerstner and Kistler, 2002;Markram, 2006) and/or realize its "electronic replicas" that partially imitate brain functionalities such as learning and memory (Mead, 1990;Jeong et al., 2013;Merolla et al., 2014;Qiao et al., 2015). The former generally employs computational SNNs; a vast number of spiking neurons are simulated on computers in search of their behaviors relating to neuronal representation at both low and high levels (Markram, 2006). By contrast, the latter relates physically working hardware SNNs and their components, e.g., spiking neurons and synapses, in favor of real-time interaction with environments, which is referred to as neuromorphic engineering (Mead, 1990). When emulating an SNN with a vast number of neurons, the hardware SNN largely outperforms the computational SNN in terms of runtime, given the latter's need for substantial computational resources. The larger the SNN, the greater the severity of its need for computational resources. The hardware SNN is thus perhaps a good solution to this practical problem, if the artificial neurons and synapses capture their biological counterparts with high precision (Indiveri et al., 2011;Azghadi et al., 2014). The components with limited precision-only capturing the essence of their biological counterparts-can be engaged in neuromorphic systems that are endowed with several brain functionalities such as spatiotemporal recognition despite the component-wise disparity in detailed behavior (Eliasmith and Anderson, 2004;Eliasmith et al., 2012).
Essentially, neurons in a biological network communicate by spikes. The membrane potential of each neuron rises amid incident presynaptic spikes that cause excitatory postsynaptic currents (EPSCs) through the dendrites. That is, the membrane integrates the EPSCs until the membrane potential reaches a threshold for spiking. This procedure is referred to as integrate-and-fire (IF; Burkitt, 2006). This IF procedure lays the foundations of computational neuron models, e.g., the leaky IF neuron (LIF; Burkitt, 2006), Hodgkin-Huxley neuron (Hodgkin and Huxley, 1952), and Izhikevich neuron models (Izhikevich, 2003), and the corresponding hardware models (Mead, 1989;Indiveri et al., 2011;Lim et al., 2015). Among the models, the LIF neuron is one of the most widely used models in light of its simplicity.
In view of real-time interaction with physical environments, it is desirable that the hardware neuron spikes at a rate similar to that of the biological neuron (ca. <100 Hz) given that each spike consumes a certain amount of power. The higher the activity in a given period of time, the more power the neuron consumes. Toward this end, the interspike interval (ISI) between neighboring spikes in time reaches a few tens of milliseconds, which requires a comparable R-C time constant within the framework of the LIF neuron. To put it precisely, a linear low-pass filter, i.e., integrator, in the LIF neuron needs to be endowed with a cutoff frequency below the minimum activity of the biological neuron. Signal integration can be realized in different integrators, e.g., Tau-cell (Edwards and Cauwenberghs, 2000), the subthreshold log-domain integrator by Arthur and Boahen (2004), and a differential pair integrator (Bartolozzi and Indiveri, 2007), and they are nicely reviewed in a paper written by Indiveri et al. (2011). For these integrators, a capacitor causes a delay in the response to an input signal so that the capacitance significantly alters the time delay, partly akin to an R-C delay in a simple R-C circuit.
The FG-based metal-oxide-semiconductor field-effect transistor (MOSFET), FG-MOSFET for short, is one of the most successfully commercialized nonvolatile memory bits in flash memory (Jeong et al., 2012). Remarkable progress in flash memory technology has been made, ranging from the charge trap flash as a variation of the FG-MOSFET to vertical NAND memory. The high maturity level of flash memory technology offers great opportunities for neuromorphic engineering; in particular, FG-MOSFETs are promisingly utilized as programmable synapses that work as local memories within a neuromorphic circuit (Hasler et al., 1994;Gordon et al., 2004;Tenore et al., 2006;Brink et al., 2013;Ramakrishnan et al., 2013). To date, diverse FG-MOSFET-based synapse circuits have been proposed with different precision; the simplest case is the singletransistor synapse device that can maintain the programmed synaptic weight for sufficiently long time periods and implement the spike-timing-dependent plasticity protocol (Hasler et al., 1994;Ramakrishnan et al., 2013). In addition, FG-MOSFETs are also employed as the core part of a synapse circuit (Tenore et al., 2006). Although synapse circuits containing FG-MOSFETs are diverse, it is common that the FG-MOSFETs are responsible for the memory of a programmed synaptic weight.
In this study, we propose an LIF neuron circuit based on a floating-gate (FG) integrator as a replacement for a capacitor integrator. Compared with FGs in synapse circuits, the role of an FG in this type of integrator is counterintuitive given that the FG is deliberately designed to retain the charge on the FG for a few seconds, at most. This poor charge retention is not acceptable in the FGs in synaptic circuits. Circuit simulations were conducted using LTspice IV in support of the proposed circuit. The kinetics of filling the floating gate with charge (charging) and emptying it (discharging) resembles the charging and discharging of a capacitor. However, a significant difference lies in the mechanism for charging and discharging. Charge transfer into and out of the FG is mainly determined by area-independent properties of the tunnel barrier, e.g., barrier height and thickness. Thus, the characteristic time constant-corresponding to that in a capacitor-based integrator-can be tweaked irrespective of the area of the FG, unlike the capacitor-based integrator. As a result, the circuit has excellent potential for scalability and very low power consumption.

Circuit Simulations
The circuit simulations were performed using LTspice IV. The LIF neuron circuit was designed by adopting 65nm complementary metal-oxide-semiconductor (CMOS) technology that was implemented by using the BSIM 4.6.0 model (a built-in model in LTspice IV; Dunga et al., 2006). The parameters for all devices in this work can be found in Table 1. Quantum mechanical elastic tunneling through the tunnel barrier in a tunnel junction is a key phenomenon in the FG integrator; we utilized the tunneling equation included in the BSIM 4.6.0 model (Cao et al., 2000;Lee and Hu, 2001;Dunga et al., 2006). The tunneling equation is based on the Fowler-Nordheim tunneling within the framework of the Wentzel-Kramers-Brillouin approximation. The tunneling equation in the BSIM 4.6.0 model is semi-empirical with regard to the use of an auxiliary function that improves the accuracy of the original Fowler-Nordheim tunneling equation (Ranuárez et al., 2006).

Noise Implementation
Consecutive random number generation is required for simulating time-varying noise to be applied to the FGLIF neuron circuit. White voltage noise, e.g., thermal voltage noise, was simulated by generating an identical and independently distributed (i.i.d.) random number whose probability follows a normal distribution. The probability distribution function (PDF) is centered at zero with a standard deviation corresponding to the root-mean-square (RMS) amplitude of voltage noise ( V RMS ). A new random number was repeatedly generated at each time bin ( t) from this PDF. The gate terminals of all MOSFETs in the neuron circuit were subject to such white noise. Burst noise and flicker noise as a group of individual burst noises are nonwhite noise. Burst noise in an n-channel MOSFET, nMOS for short, is estimated to originate from repeated localization and delocalization of electrons by traps at the gate oxide/semiconductor interface (Hung et al., 1990). Both localization (trapping) and delocalization (detrapping) are stochastic and renewal processes with regard to the exponentially decaying PDF with the duration of an empty trap (τ t ) and a filled trap (τ det ) (Yonezawa et al., 2013). That is, the interaction between an electron and a trap is a Poisson process. First we defined the electron-trapping (detrapping) rate that evaluates the number of trapping (detrapping) events per unit time as r t = 1/τ t (r det = 1/τ det ). A uniform random number in the range between 0 and 1 was generated at each time step and compared with r t · t to determine the occurrence of an electron-trapping event; the electron is trapped at the time step if the random number is smaller than r t · t. The same holds for an electrondetrapping event except that the random number is compared with r det · t. The time bin t was sufficiently small (100 µs). For simplicity, it was assumed that r t = r det .
The traps were assumed to be neutral if they were empty and located at the gate oxide/Si interface. Additionally, each trap was assumed to interact with only one electron. Each filled trap induces a change in the flat band voltage on average △V fb by △V fb = α · t ox · N 0.6 a / √ L eff · W eff , where α, t ox , N a , L eff , and W eff denote an empirical constant, gate oxide thickness, acceptor density in the channel, and effective channel length and width, respectively (Fukuda et al., 2007). For all MOSFETs, α was set to 1.5 × 10 −12 (Fukuda et al., 2007) and N a to 1.7 × 10 17 cm −3 (default value in the BSIM 4.6.0 model).
V fb is generally distributed following an exponential PDF; the abovementioned △V fb denotes the expected, i.e., mean, value given the exponential PDF (Fukuda et al., 2007). In this regard, each trap was endowed with a particular random V fb value that was obtained from the exponential distribution. Note that V fb upon an electron-trapping event is positive given the appearance of a negative point charge at the gate oxide/Si interface.
Given the stochastic electron-trapping and detrapping processes, V fb fluctuates with time following a Poisson process. The fluctuation in V fb ( V fb ) consequently alters the drain current (I d ) at a given gate voltage (V g ) with regard to the consequent change in the threshold voltage V th ( V th ), and V th = V fb . The transconductance g m is a function of the difference between V g and V th0 + V th , i.e., where V th0 denotes V th in the absence of a trap. Imposing V fb on V fb is thus equivalent to V g subject to a fluctuation by − V fb with a noise-free V th , i.e., V th0 . In our circuit simulations, the stochastic change in V fb and the resulting change in g m were simulated by changing V g by − V fb at each time step. The aforementioned thermal noise was applied to V g on top of this burst noise.
To simulate flicker noise, a group of n traps was assumed to simultaneously interact with electrons, and the interaction of each trap with an electron was independent of the others. The noise generation algorithm was analogous to the abovementioned burst noise generation except that the independent interactions of all n traps with electrons were simultaneously considered. The occurrence of trapping or detrapping at each trap at each time step was determined by comparing r t · t or r det · t with an i.i.d. random number. r t for each trap was randomly chosen as for the abovementioned single-trap case. This comparison was repeated n times with n i.i.d. random numbers.
The same method was applied to p-channel MOSFETs, pMOS for short, regarding the interaction between a trap and a hole (rather than an electron). The donor density in the channel (N d ) was 1.7 × 10 17 cm −3 , which is the default value in the BSIM 4.6.0 model. The main difference from the nMOS case lies in the charge of a filled trap, which is positive and negative for a pMOS and nMOS, respectively. Consequently, V fb for a pMOS upon a hole-trapping event is negative because of the appearance of a positive point charge at the gate oxide/Si interface, which is the opposite of the nMOS case. Thus, the equivalent change in V g is also the opposite of the nMOS case.

Circuit Configuration
The proposed floating-gate-based leaky integrate-and-fire (FGLIF) neuron circuit is depicted in Figure 1. The circuit consists of 12 MOSFETs (M2-M13), a single FG transistor (M1 + C FG ), and a capacitor C1. The FG transistor has separate terminals MT1 and MT2 (tunnel junctions) for programming charge in the FG through quantum mechanical tunneling. Such an FG transistor is often referred to as a synapse transistor (Diorio et al., 1998;Rahimi et al., 2002). As shown in Figure 1, the circuit is divided into four subcircuits on functional grounds: (a) charge integrator, (b) non-inverting common source amplifier (Amp1), (c) non-inverting common-source amplifier (Amp2) with positive feedback, and (d) polarity inverter.
To begin, it is worth noting the synapse transistor ( Figure 1A) that plays the key role in the proposed LIF neuron circuit. The additional terminals on MT1 and MT2 exclusively control the charge in the FG by means of quantum mechanical tunneling through the tunnel barriers in MT1 and MT2 given the use of a relatively thick gate oxide layer (2.5 nm) in FG transistor M1, which hinders the charge transfer through it. To put it precisely, the FG charge is controlled in a vertical tunnel junction of electrode-barrier-FG-barrier-electrode, following the prototypical synapse transistor proposed by Hasler et al. (1994). In the circuit simulator, the tunnel junctions are implemented by shorting the source, drain, and body terminals of a MOSFET, as shown in Figure 1A. Presynaptic currents are incident on the shorted terminal. All dielectrics in use in the circuit are SiO 2 unless otherwise stated.

Integrator Circuit
The integration in the circuit is realized by the FG integrator in Figure 1A. A charge is injected into the FG upon an incident presynaptic current on the tunnel junction and temporarily remains trapped. FG potential V FG evolves in due course, which consequently changes the channel conductance of M1. V FG thus parameterizes the integration. The injected charge simultaneously decays with time amid the presynaptic current injection, implying leaky integration that lays the foundation of LIF behavior in the FGLIF neuron. The retention time of an FG charge is mainly determined by the tunnel barrier thickness d tun_ox and area, and the capacitance of C FG . The thick gate oxide of M1 barely intervenes in the charge ejection on the timescale of our interest. In contrast to an FG transistor as a nonvolatile memory element, our approach aims at the active use of charge-ejection dynamics; therefore, a charge retention time of approximately several seconds is desired. Accordingly, the circuit design and programming voltage are in need of tweaks. Several device parameters of the FG integrator in this work are listed in Table 1. The parameters in Table 2 were used for the circuit simulations unless otherwise stated.
The synapse transistor has an advantage of employing multiple input tunnel junctions, enabling spatial integration (Polsky et al., 2004) that denotes the simultaneous integration of synaptic currents through different synapses. Figure 2A displays an FG integrator with n identical input terminals (MT1 1 -MT1 n ). For n = 10, a time-varying V FG in response to an incident spike on a single terminal at 0 s was simulated with the other nine terminals being grounded. For comparison, the same simulation was conducted for n = 1. The input spike amplitude (V in ) and width (t sp ) were 0.5 V and 10 µs, respectively. Figure 2B relates the simulation results that uncover a decaying V FG with different time constants, i.e., relaxation times, which depend on n and d tun_ox . The relaxation time is defined as the requisite time for V FG to reach 1/e of V FG at 0 s. The higher the number of input terminals, the shorter the relaxation time since each terminal works as a charge leakage path. Notably, a time constant of ca. 2.7 s for n = 1 is significantly reduced to 0.3 s for n = 10, as seen in Figure 2B. A workaround solution to such a reduction is to make use of thicker tunnel barriers, which offers a larger relaxation time ( Figure 2B). The use of a thicker tunnel barrier trades off V FG at 0 s for a larger relaxation time in light of the difficulty in charge injection upon spike arrival. Thus, one should carefully choose the tunnel barrier thickness that reconciles the charge relaxation (ejection) kinetics with the charge injection kinetics.

Amplifiers
MOSFETs M2-M5 in Figure 1B form a signal amplifier (Amp1) of two inverting common-source stages with pMOS loads. The input into Amp1 (V amp_in ) is controlled by a voltage drop across the channel of M1, which is determined by V FG . That is, V FG determines the output of Amp1. The output is subsequently relayed to the next amplifier (Amp2) in Figure 1C. The voltagetransfer characteristic (VTC) of each stage in Amp1 is controlled by constant gate voltages (V g1 and V g2 ); they are important in designing the output spike width and spiking threshold. We will set aside this issue until Section Adjustment of Circuit Parameters. The signal exiting from Amp1 (V amp_out ) enters Amp2, which directly elicits a spike at output terminal V out when V amp_out exceeds the transition region in the VTC of Amp2. The transition region is determined by two identical inverters (M6-M7 and M8-M9). The midpoint voltage, where V out = V amp_out , is taken as a threshold voltage of Amp2 for spiking. Capacitor C1 realizes a positive capacitive feedback to the input to Amp2, allowing the output to remain high until the reset of the FG integrator. The reset occurs in rapid succession following the onset of spiking at V out by negative feedback that is achieved by the polarity inverter in Figure 1D.

Polarity Inverter
The FG integrator is in need of a reset in order to complete a single spike. The reset is equivalent to emptying a charge in the FG by means of an electric field. The application of a negative voltage to terminal MT2 lets the previously injected charge vanish. Thus, a subcircuit that inverts V out and relays it to the FG integrator is necessary. The subcircuit in Figure 1D, in conjunction with negative V dd (V dd− ), flips the polarity of V out , resulting in a negative output at V inv in Figure 1D. To highlight this polarity inverter, the subcircuit is separately illustrated in Figure 3A, and the simulated VTC at a V dd− of −0.5 V is plotted in Figure 3B. Note that V in in Figure 3A corresponds to V out in Figure 1. The VTC evidences a polarity reversal for V in larger than the midpoint voltage. A negative V inv pulse is accordingly elicited from the polarity inverter in response to a positive V in pulse (0.5 V in amplitude and 25 µs in width), as seen in Figure 3B.
Provided that such a negative voltage pulse resets the FG integrator, V FG consequently falls below zero. This reset process continues until V FG becomes sufficiently low to let an input to either amplifier fall below the threshold for amplification.

Dc Input Mode
First, we verify the spiking dynamics of the FGLIF neuron circuit (n = 1) under a constant voltage, which is equivalent to controlled neurophysiology experiments. Applying a constant voltage to MT1 continuously elevates V FG (integration) in light of the positive charge injection into the FG. With that said, the detailed balance (Riggert et al., 2014) eventually reconciles the charge injection with the ejection at a particular V FG level; therefore, the rate of a V FG increase largely declines when it is close to this level. By contrast, an R-C integrator maintains such a balance through charging on C and simultaneous discharging through R. Figure 4B shows the V FG variation in time at 0.26 V amid output spiking. V FG rises in the first place, and thus so does the channel conductance of M1. V amp_in and the resulting V amp_out consequently increase until V amp_out reaches the threshold (ca. 0.29 V) for high V out through Amp2, as plotted in Figure 4C. The high V out is temporally maintained in view of the positive feedback through C1. The high V out simultaneously triggers the polarity inverter that activates negative feedback to the FG integrator, resetting the FG integrator. V amp_out therefore falls below the threshold, leading to the termination of the high V out and negative feedback. This procedure produces a single spike and is repeated for the next spike within the ISI. The reset rate determines the spike width: the faster the reset, the narrower the spike width. This relationship will be addressed in detail in Section Adjustment of Circuit Parameters. The simulation with the parameters in Table 1 uncovered an output spike width of 25 µs and an output activity a out (spiking frequency) of ∼23 Hz ( Figure 4D).
The neuronal gain function is the substrate of encoding neuronal information (Gerstner and Kistler, 2002;Eliasmith and Anderson, 2004); different inputs are encoded to represent distinguishable outputs. In an attempt to verify a gain function of the FGLIF neuron, the neuronal activity was evaluated at different constant voltages (Figure 4E). A higher input voltage significantly speeds up FG charging within the ISI, reducing the ISI to a large extent. Thus, the activity largely increases with the input voltage. Figure 4E also uncovers a threshold input for spiking (60, 100, and 190 mV for V g2 = 0.60, 0.65, and 0.70 V, respectively), which is determined by the threshold of Amp2. Furthermore, there exists a minimum activity of ∼0.2 Hz,  and it features the type-II excitability of the Hodgkin-Huxley model (Dayan and Abbott, 2001). The minimum activity is fairly negligible. Note that the exponential change in activity upon voltage (Figure 4E) arises from the tunnel current that exponentially varies upon voltage (Jeong and Hwang, 2005;Soni et al., 2014).

Spiking Input Mode
A spiking input mode realizes practical circumstances for the operation of an FGLIF neuron in an SNN. As a whole, the response of the FGLIF neuron is comparable to the dc input mode, although there are differences to some extent. The subcircuit-wise responses to an input spike train (activity: 100 Hz; spike amplitude: 0.42 V; spike width: 25 µs) are plotted in Figure 5 in the same order as Figure 4. The main difference between the two modes lies in the integration: the input spikes cause stepwise evolution of V FG , unlike the former case ( Figure 5A). However, irrespective of an input type, the FG integrator successfully features leaky integration. Analogous to the dc input mode, V amp_in is amplified through Amp1 (Figure 5B), and a spike is elicited from V out when V amp_out crosses the threshold of Amp2 ( Figure 5C). Likewise, the output activity differs for different spike amplitudes and activities, as shown in Figure 5D; the larger the amplitude or/and activity, the more frequently the neuron spikes. That is, when information transmission between pre-and postsynaptic neurons is invoked in an SNN, the postsynaptic FGLIF neuron is able to represent the presynaptic neuron's activity, i.e., input activity for the postsynaptic neuron, by outputting a distinguishable activity.

Adjustment of Circuit Parameters
The proposed circuit provides a means of tweaking neuronal behavior such as relaxation time, spiking threshold, and spike width and amplitude. Recalling the change of the relaxation time upon the tunnel barrier thickness (Section Circuit Simulations), a thicker tunnel barrier, e.g., 1.5 nm for n = 10, is a priori preferred in favor of a relaxation timescale that is biologically plausible. The consequent decrease in maximum V FG can be compensated for by increasing the spike width. For instance, for d tun_ox = 1.5 nm, the use of a wider spike (100 µs) raises the maximum V FG by approximately one order of magnitude, per our simulation (not shown).
In addition, gate voltages V g1 and V g2 of the loads in Amp1 alter the VTC. For instance, Figure 6A shows the VTC of Amp1 for three different V g2 values at the same V g1 (0.7 V) where significant changes in the VTC are seen. The same holds for V g1 as shown in Figure 6B; however, the VTC merely shifts relying on V g1 . Provided that high V out (spiking) is triggered only if V amp_out reaches the threshold of Amp2 (denoted by a dashed line in Figure 6A), V amp_in for spiking substantially varies upon V g2 . In Figure 6A, a higher V g2 leads to a higher V amp_in for spiking; therefore, spiking requires a higher V FG . A higher V FG is in need of a longer integration time at a given input voltage in the dc input mode, or equivalently at a given input activity (if sufficiently high to evoke a spike) in the spiking input mode. Thus, output activity a out declines with V g2 (Figure 6C). On the same grounds, a higher V FG requires a higher V in to output the same activity. As a consequence, the threshold for spiking in the dc input mode increases with V g2 , as shown in Figure 6D. Therefore, the neuronal gain function can be easily modified by tweaking V g2 .
Triggering the positive feedback through C1 elevates V amp_out over the threshold of Amp2, as seen in Figures 4C, 5B; the overshoot (ca. 65 mV in this work) is mainly determined by the capacitance of C1 and V dd+ . The overshoot is independent of V g2 . Then V amp_in immediately declines upon the onset of the negative feedback to the FG integrator through the polarity inverter. V amp_in eventually falls below the threshold of Amp2. The requisite time for this process is equivalent to the spike width. The V amp_out -V amp_in relations in Figure 6A indicate that the higher V g2 is given, the larger decrease in V amp_in ( V amp_in ) needs to be made by the negative feedback to drag V amp_out below the threshold. V amp_in is elucidated in the inset of Figure 6E, where the VTCs in Figure 6A are zoomed in. Notably, the requisite V amp_in increases with V g2 , and thus it takes longer for a higher V g2 to decrease V amp_out by 65 mV, i.e., to reset the FG integrator with the same negative V inv . As a consequence, a higher V g2 offers a wider spike, as shown in Figure 6F. In addition, given that a higher |V dd− |evokes a higher |V inv |, resetting the FG integrator takes less time with a higher |V dd− |, rendering the output spike width narrower ( Figure 6F).

Power Consumption
Regarding the principles of neuromorphic engineering, low power consumption is strongly desired. Toward this end, we evaluated the power consumption of the FGLIF neuron that elicits different output activities amid the application of a constant input voltage. The average power consumption was acquired for various output activities by evaluating the consumed energy during the period of a single spike and dividing it by the period. The results are plotted in Figure 7, in which the average power consumption is gently proportional to the output activity. Notably, the proposed FGLIF neuron circuit consumes power less than 30 pW in the entire activity range. This is mainly ascribed to the subthreshold operation of Amp1 and Amp2, allowing a low current flow through the channels in series.
Power consumption of the proposed circuit was compared to the values achievable in the alternative VLSI neuron designs, which are listed in Table 3. The FGLIF neuron circuit provides power consumption that is several orders of magnitude lower than those of other models, while the number of transistors in use is comparable. Of course, a precise comparison between the circuits is still difficult since power consumption also strongly depends on several other factors such as CMOS technology in use, neuron spike width, and firing rate. Therefore, the results in Table 3 provide only an approximate overview and can be expected to change.

Spiking in the Presence of Noise
The operation of the FGLIF neuron circuit appears to be markedly susceptible to noise because of the subthreshold operation of the MOSFETs in the circuit. Thus, it is important to identify the effect of noise on the operation of the neuron circuit and analyze it in comparison with its biological counterpart. Types of noise in a MOSFET are (i) thermal noise, (ii) shot noise, (iii) burst noise, and (iv) flicker noise. The first two are white noise, whereas the last two are nonwhite noise whose power spectral density (PSD) relies on frequency (Chong and Sansen, 2013;Hamilton et al., 2014). The thermal voltage noise across a MOSFET channel is analogous to that across a resistor, and its PSD is S v = 4kTR d (Nyquist, 1928). Here R d denotes the resistance of the channel. In the FGLIF circuit, the thermal noise in each CMOS stage (a pair of pMOS and nMOS channels) is filtered by the capacitance of the following stage. Therefore, the thermal noise endows each voltage input node with V RMS : where k b , T, and C denote the Boltzmann constant, temperature, and equivalent MOS capacitance of the following stage, respectively. Note that the MOS capacitance relies on the gate oxide capacitance and gate voltage. Given the subthreshold operation of the MOSFETs in the neuron circuit, C is smaller than that of above-threshold-working MOSFETs, and thus the thermal noise effect is perhaps prominent with regard to Equation (1). For simplicity, C was evaluated at the average V g values that are applied to the input of each CMOS stage during the circuit's operation. V RMS was evaluated at each node that precedes a given capacitance value, and it varied from 0.8 to 5.4 mV for different nodes. This thermal noise was taken into account in the following circuit simulations, as detailed in Section Materials and Methods. Additionally, it should be mentioned that Equation (1) is in fact not limited to thermal noise but is rather universal for the entire white noise of the system in the presence of a filtering capacitor (Sarpeshkar et al., 1993). As a result, no extra terms are needed to evaluate the shot noise in the circuit. Burst noise (also known as random telegraph noise) appears to markedly affect the operation of the neuron circuit, given the fluctuation of V fb , i.e., V fb , upon the interaction of a charge trap with an electron. The contribution of each trap interacting with an electron to △V fb differs for the MOSFETs in the circuit owing to the different channel areas and gate oxide thicknesses (see Section Materials and Methods). In the following simulations, △V fb for each MOSFET was evaluated, and the fluctuation of V g (equivalent to V fb that was randomly generated from an exponential distribution function with △V fb ) was applied to each MOSFET following the Poisson process that is elucidated in Section Materials and Methods. The same held for M1 (FG-MOSFET) except that V fb was imposed on V FG rather than V g . The electron-trapping rate r t was also an i.i.d. random variable (see Section Materials and Methods).
The topmost panel in Figure 8A shows a typical fluctuation in V g for nMOS (channel area: 120 × 60 nm 2 ; gate oxide thickness: 2.5 nm) attributed to a single trap at the gate oxide/Si interface.
V fb was 1.0 mV. V g is thus toggled between 0 and −1.0 mV upon the interaction between the trap and an electron; V g is −1.0 mV when the trap is filled with an electron, and V g is 0 otherwise. The corresponding PSD is plotted in Figure 8B, in which the PSD dispersion in the double logarithmic plot follows a power low with an exponent of −2 (Lorentzian PSD). The same holds for pMOSs with the same parameters, except that V g is toggled between 0 and 1.0 mV, and V g is 1.0 mV when the trap is filled with a hole.
The number of traps underneath the gate oxide is not necessarily one. Interfacial trap density D it evaluates the average number of traps per unit area in the n-channel area; the larger Average trapping and detrapping times for single-trap case (upper graph) were chosen to be 10 ms each, while in other cases they were taken from exponential distribution with a mean value of 10 ms. V fb owing to a single trap was also taken from exponential distribution with a mean value of 1.0 mV. (B) Noise PSD, evaluated for the temporal patterns in (A).
D it , the more traps are likely present at the interface. The number of traps (7, 14, and 72 traps) markedly alters the fluctuation in V g , as shown in Figure 8A. An i.i.d. random V fb and r t were assigned to each trap; both were drawn from exponential distribution functions with △V fb differing for different MOSFETs and a τ t of 10 ms, respectively. Additionally, each trap was assumed to interact with a single carrier. The method of noise generation is detailed in Materials and Methods. As shown, the V g deviation from 0 increases with the number of traps because V fb proportionally increases with the number of filled traps. Notably, the PSD becomes close to flicker noise with the number of traps; for instance, 72 traps (orange PSD in Figure 8B) lead to a PSD approximately following an exponent of -1 in the double logarithmic plot. This is a consequence of the presence of multiple Lorentzian PSD functions (72 in total), i.e., superposition of these many PSD functions results in a 1/f PSD function (Campbell et al., 2009). Burst noise is thus a subset of flicker noise with regard to their origins. Long-channel MOSFETs often exhibit flicker noise attributed to the probable large number of traps at a given D it (Uren et al., 1985). By contrast, deepsubmicron MOSFETs likely include a few traps (or even a single trap); therefore, burst noise is predominantly observed (Hung et al., 1990).
Eventually, the FGLIF neuron circuit was simulated taking into account the aforementioned types of noise. The circuit simulation was done for both dc input mode (V in : 0.26 V) and spike input mode (a in : 100 Hz, V in : 0.42 V) for direct comparison with the behaviors without noise shown in Figures 4, 5. Figure 9 displays the simulation results, including output spikes in time and ISI distribution. Note that D it for all MOSFETs was set to 10 11 cm −2 in the simulations. The ISI (τ ) distribution apparently arises from the present noise as compared with the perfect periodicity of spikes (single ISI) for the noise-free neuron circuit. The ISI (τ ) histogram for both input modes can be fitted well to a Gamma distribution function given by where k and θ are fitting parameters that determine the shape and scale of the distribution, respectively (Maimon and Assad, 2009). As shown in Figure 8, the noise characteristic markedly varies upon the number of traps; therefore, the ISI distribution may change accordingly. To identify this effect, additional ISI histograms were obtained with different D it values (10 11 , 2×10 11 , and 4 × 10 11 cm −2 ), and the results are shown in Figure 10. The higher number of traps at the interface, the more widely the ISI is distributed. The decrease in the shape parameter k denotes an increase in spiking irregularity (Maimon and Assad, 2009), which is captured by the ISI widening. Note that in these simulations the contribution of each noise source to the overall noise effect cannot be distinguished mostly due to the physical origin of flicker and burst noise as mentioned earlier. Nevertheless, the power spectral density arising from interfacial charge traps becomes close to flicker noise with the number of charge traps (see Figure 8B), and thus the change in the ISI distribution with trap density in Figure 10 most likely reflects the distributional change upon the transition of a noise mechanism from burst-like to flicker-like noise. This transition widens a range of trapping and detrapping probabilities that are proportional to trapping and deptrapping rates as detailed in Section Materials and Methods. Consequently, a wide range of such rates is intertwined in the overall noise dynamics. Therefore, spiking regularityparameterized by k-increases with trap density.
To identify the relative contributions of thermal and burst noise to the overall ISI distribution, further simulations were conducted by controlling the noise sources. Three cases were considered: thermal noise only, burst noise (subset of flicker noise) only, and simultaneous thermal and burst noise. Each case was examined from 10 independent simulations. The resulting ISI distributions (fitted to Gamma distribution functions) are shown in Figures 11A-C. As such, the thermal noise is determined by capacitance (see Equation 1) so that it is almost invariant through the trials. This is featured by the negligible trial-to-trial variation in the ISI distribution ( Figure 11A). Notably, the average ISI shifts toward a lower value due to the thermal noise as seen in comparison with the noise-free case (ca. 43.5 ms) indicated by a dashed vertical line. By contrast, the burst noise markedly alters the ISI distribution upon trial ( Figure 11B) because V fb and r t values were sampled at random for each trial from exponential distributions (Yonezawa et al., 2013). The center of the distribution for each trial is scattered around the ISI of the noise-free case. In the present of simultaneous thermal and burst noise, the effects are superimposed ( Figure 11C); the scattered positions of distributional centers relate to the burst noise, and their shift below 43.5 ms relates to the thermal noise. The noise-sensitive ISI distribution is mostly dictated by the noise in Amp1 (Figure 11B) whose input (V amp_in ) directly evokes a spike. The thermal noise endows V amp_in with a fluctuation around the noise-free V amp_in so that the output from the input stage (M2-M3), i.e., V M3 in the inset of Figure 11D, fluctuates around the noise-free VTC (gray zone in Figure 11D). This fluctuation, in turn, affects the input into the following stage (M4-M5) in conjunction with its own noise. A typical reduction in ISI due to thermal noise is shown in Figure 11G. The output overshoot accelerates spiking (orange line) that precedes the noise-free spiking (gray line) by ∼10 ms. Therefore, the thermal noise generally shorten the ISI.
The effect of burst noise on Amp1 differs for pMOS and nMOS. For pMOS, the trapped holes at the gate oxide/Si interface leads to a negative shift in V fb , i.e., V fb < 0; therefore, when the burst noise in the pMOS is solely present in the input stage, the VTC shifts as a whole as shown in Figure 11E. Such a shift causes a reduction in Vamp_in to output the same V M3 as for the noise-free case so that the ISI is reduced-similar to the thermal noise effect. An example is shown in Figure 11H. By contrast, for nMOS, the electrons trapped at the channel interface elevate V fb , i.e., V fb > 0, indicating a right shift in the VTC as a whole ( Figure 11F). Consequently, Amp1 needs higher V amp_in than the noise-free case to output the same V M3 as for the noise-free case, implying larger ISI. An example of this case is plotted in Figure 11I.

DISCUSSION
Recall that unlike capacitor-based integrators, the FG integrator in the proposed FGLIF neuron has the characteristic relaxation time defined by quantum mechanical tunneling dynamics through the tunnel barrier. The different basis of charge integration largely mitigates a severe need for high capacitance in favor of a biologically plausible timescale. In order to highlight the scalability of the proposed FGLIF neuron circuit (particularly the FG integrator), the FG integrator was compared with a switched-capacitor integrator comprising n MOSFET switches (MR1-MRn) and one capacitor C mem . The switched-capacitor integrator is illustrated in Figure 12A. Note that the MOSFETs have the gate and drain shorted in order to realize "fast charging and slow discharging." They are switched on upon the application of a voltage pulse to the shorted terminal.
All MOSFETs in this integrator were of the same size (channel length/width: 60/120 nm) as the tunnel junctions and FG transistor M1 in the FG integrator. The circuit simulation results indicate a need for remarkably high capacitance-more than three orders of magnitude higher than that of C FG (6 fF)to achieve a relaxation time of a few seconds for the FG integrator (compare Figure 12B with Figure 2B). For a fair comparison, the input voltage pulse was identical to that of the FG integrator shown in Figure 2 (amplitude: 0.5 V, width: 10 µs). Furthermore, the use of multiple terminals (n = 10) significantly reduces the relaxation time by approximately one order of magnitude in light of a decrease in the equivalent resistance owing to the MOSFETs being in parallel ( Figure 12B). The same holds for the FG integrator as addressed in Section Circuit Simulations, and a relaxation time of a few seconds was recovered by introducing a slightly thicker tunnel barrier. By contrast, the switchedcapacitor integrator requires a higher capacitance to avoid such a large decrease in the relaxation time, e.g., 90 pF to endow the integrator with a relaxation time of ∼2.5 s, as shown in Figure 12B. However, the high capacitance value retards not only the discharging (relaxation) but also the charging; therefore, the V mem maximum (ca. 24.8 mV) was not reached during the period of a single spike.
Such high capacitance in this simple integrator is hardly affordable in integrated circuits, delimiting the scalability. It is revealed that realizing a capacitance of a few pF requires a few hundred µm 2 , which cannot fit in with the framework for scaling down. As addressed in Section Circuit Simulations, C FG of 6 fF in capacitance was placed on M1 (channel length/width: 60/120 nm), and thus the FG integrator scheme offers great scalability, which is a definite advantage over switched-capacitor integrators. Of course, several strategies for reducing the capacitor area are likely available, e.g., introducing high-k dielectrics, or HfO 2 and/or three-dimensional capacitors (Kim et al., 2010). However, such attempts may cause additional complexity in chip fabrication and a consequent fabrication cost that may outweigh the benefits. Alternatively, maximizing the resistance of the switch in a switched-capacitor integrator equally enables a long relaxation time, as demonstrated by Noack et al. (2015). Another strategy based on capacitor-based integration by Qiao et al. (2015) accomplished a time constant of a few tens of milliseconds using a 1 pF capacitor. Nevertheless, our FG-based strategy may be a potential alternative to the capacitor-based integration for further scaling down neurons.
One of the main concerns for flash memories is reliability, particularly in the endurance of the FG transistor. Poor endurance-although sufficient for flash memory applications with a write endurance of 10 4 -10 5 times-is mainly ascribed to a high write voltage in exchange for better data retention FIGURE 11 | ISI variation upon noise-type. Output ISI distributions in the presence of (A) only thermal noise, (B) only burst noise, and (C) simultaneous thermal and burst noise at a constant dc input of 0.26 V. The dashed vertical lines at 43.5 ms indicate the ISI of noise-free spiking. The VTC of the input stage (M2-M3) of Amp1-shown in the inset-in the presence of (D) only thermal noise and only burst noise in (E) pMOS (M2) and (F) nMOS (M3). The shaded regions denote VTC variation ranges. The evolution of output of Amp1 (V amp_out ) for cases of (D-F) is exemplified in (G-I), respectively. (Cappelletti and Modelli, 1999). Given the circumstances for the operation of the FG integrator (receiving a train and/or burst of a number of spikes), the poor endurance is perhaps a significant obstacle. Low neuronal activity (as low as biological neurons) is preferred partly on these grounds. Fortunately, unlike with flash memory, we deliberately allow data (charge on the FG) loss for a relaxation time of a few seconds, so that the requisite write and erase voltages are much lower, i.e., ±0.5 V in the proposed circuit. This low voltage is mostly applied to the tunnel junction; the voltage across C FG is merely 47 mV (maximum V FG shown in Figure 4B) at most, and the gate oxide is subject to even less voltage with regard to the additional voltage drop in the Si channel underneath. The simulation results show ∼90 electrons that tunnel through the gate oxide in a single spiking period, i.e., one output spike (amplitude: 0.5 V, width: 25 µs) and one ISI, which is equivalent to a charge density of ∼2×10 −7 C/cm 2 . Thus, such low voltage likely alleviates the burden on the FG, rendering the FG fairly endurable (Wann and Hu, 1995).
Significant improvement in the reliability of flash memory occurred by replacing the standard silicon FG with an insulating SiN x one, forming so-called silicon-oxide-nitride-oxide-silicon (SONOS; Chan et al., 1987). The insulating SiN x tolerates shorting paths embedded in the tunnel barriers to a greater degree than the standard SOSOS (silicon instead of SiN x in SONOS; Chan et al., 1987). However, provided that the synapse transistor stretches the FG to the separate tunnel junctions, the conductance of the FG is of significant concern. To put it precisely, the highly resistive SiN x FG barely alters V FG in M1 upon a charge transfer through the separate tunnel junctions. The standard SOSOS is therefore suitable for the FG integrator. The aforementioned low write and erase voltages that suffice for the FG integrator likely support the reliability of SOSOS.
As such, the number of the interfacial charge traps is most likely proportional to the channel size, and the effect of a single trap on V fb is predicted to become more significant as the channel size shrinks (Fukuda et al., 2007;Miki et al., 2012). Relaxation of V mem -parameterizing integration-with time for different n's (1 and 10) and capacitance values of C mem (9 and 90 pF). An input spike (V in = 0.5 V, t isp = 10 µs) was applied to a single tunnel junction at 0 s and the others were grounded.
The continuum-based model in this study may not precisely elucidate the effect of interfacial traps on V fb in a state-ofthe-art nanoscale MOSFET. However, discrete models that are more suitable for dealing with a few traps also indicate the same tendency as for the continuum model (Fukuda et al., 2007;Miki et al., 2012). Experimental studies have pointed to burst noise as a dominant type of noise in nanoscale MOSFETs (Tega et al., 2009;Miki et al., 2012).
Noise in integrated circuits is inevitable as such, and the noise induces irregular spiking patterns (Figures 9, 10). The ISI histograms are nicely fitted to Gamma distribution functions with different fitting parameters akin to biological neurons, e.g., parietal neurons (Bair et al., 1994;Maimon and Assad, 2009). The integrated FGLIF neuron circuit in practice may differ in noise characteristics from our theoretical one. Even among integrated FGLIF neuron circuits, the difference in noise is most likely evident because the variables that relate to noise, e.g., D it , V fb , and r t , are markedly dependent on the details of fabrication methods. It is obvious that traps are incorporated in a MOSFET to some extent, and they endow the MOSFET with noise. A fundamental question arises as to whether such irregular spiking helps neural processing or is an obstacle to neural processing. Within the framework of stochastic electronics (Chen et al., 2010;Hamilton et al., 2014), such irregular spiking is necessary, and imperfections such as charge traps are required for the disorder. For instance, it has turned out that the noise given to a VLSI Hodgkin-Huxley neuron enhances signal modulation (Chen et al., 2010). Nevertheless, answering this fundamental question is beyond the scope of this paper, so we leave the question open.

CONCLUSION
We proposed an FGLIF neuron circuit, which likely achieves input-dependent output spiking activity within a biologically plausible range with a capacitor of merely 6 fF. Such low capacitance offers a great opportunity for scaling down the FGLIF neuron circuit while maintaining the activity scale. In addition, given the subthreshold operation of most MOSFETs in the circuit, spiking consumes less than 30 pW of power irrespective of spiking activity, rendering the FGLIF neuron very suitable for large-scale SNNs. In addition, the FGLIF neuron circuit is fully compatible with standard CMOS technology, which is of great benefit. Unavoidable noise in the circuit leads to distributional features of ISIs. The ISI distribution was fitted to a Gamma distribution function that has often described the ISI distribution of neocortical neurons.

AUTHOR CONTRIBUTIONS
VK conceived the idea and performed the circuit calculations together with HL, JS, and BC, GK, SK, and IK also conceived the idea and support the calculations. DJ initiated and supervised this study and wrote the manuscript. All authors discussed the results and contributed to the refinement of the manuscript.