Original Research ARTICLE
Design of a Power Efficient Artificial Neuron Using Superconducting Nanowires
- 1Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, Cambridge, MA, United States
- 2Department of Physics and Astronomy, Colgate University, Hamilton, NY, United States
With the rising societal demand for more information-processing capacity with lower power consumption, alternative architectures inspired by the parallelism and robustness of the human brain have recently emerged as possible solutions. In particular, spiking neural networks (SNNs) offer a bio-realistic approach, relying on pulses, analogous to action potentials, as units of information. While software encoded networks provide flexibility and precision, they are often computationally expensive. As a result, hardware SNNs based on the spiking dynamics of a device or circuit represent an increasingly appealing direction. Here, we propose to use superconducting nanowires as a platform for the development of an artificial neuron. Building on an architecture first proposed for Josephson junctions, we rely on the intrinsic non-linearity of two coupled nanowires to generate spiking behavior, and use electrothermal circuit simulations to demonstrate that the nanowire neuron reproduces multiple characteristics of biological neurons. Furthermore, by harnessing the non-linearity of the superconducting nanowire’s inductance, we develop a design for a variable inductive synapse capable of both excitatory and inhibitory control. We demonstrate that this synapse design supports direct fan-out, a feature that has been difficult to achieve in other superconducting architectures, and that the nanowire neuron’s nominal energy performance is competitive with that of current technologies.
The human brain has long been a subject of fascination due to the wide variety of complex operations made possible by groups of a single component—the neuron. Now, as computation needs are rapidly approaching the limits of traditional von Neumann architectures, the neuron’s unique features have led it to become a source of inspiration for new directions for the advancement of computing. Unlike conventional computing schemes, the human brain benefits from characteristics such as extensive parallelism and robustness to errors, allowing it to operate efficiently despite slow speeds on the order of a few Hz (Furber and Temple, 2007). These appealing qualities have spurred the development of technologies that use the brain as a platform for information processing, ranging from small scale modeling of single neuron dynamics to large-scale parallel computing.
At the heart of this concept are spiking neural networks (SNNs), which seek to mimic the spiking dynamics of the brain in order to encode information, with the additional benefit of possibly offering new insight into the brain’s functionality. In this scheme, spikes serve as the tokens of information, while neurons act analogously to logic gates, producing a single output in response to a combination of multiple inputs (Furber and Temple, 2007). Synapses, which connect neurons, also play a crucial role in the overall architecture by enabling learning and allowing for adaptable networks. Past approaches to SNNs vary both in degree of bio-realism and in how the spikes are implemented. While software approaches that hard-code spiking dynamics offer flexibility and precision, they are computationally expensive. As a result, other SNNs have used a hardware implementation, relying on devices with intrinsic dynamics that replicate neuron behavior as a means of reducing computation costs. Hardware approaches have been explored in a wide variety of platforms, including CMOS (Sourikopoulos et al., 2017), magnetic materials (Li et al., 2017), memristors (Querlioz et al., 2011; Thomas, 2013), and superconducting Josephson junctions (Crotty et al., 2010; Schneider et al., 2018). CMOS circuits provide large-scale integration, but suffer from high power dissipation and need many components to achieve biological realism. Memristors have characteristics which are similar to biological synapses, but need to be paired (Jeong and Shi, 2018) with spiking neurons made from CMOS circuits for SNNs and thus have similar limitations with their power dissipation. Magnetic materials can produce spiking from the spin-torque effect (Matsumoto et al., 2019), but have yet to be integrated into a full system with synapses and will most likely not be as energy efficient as superconductors. Josephson junctions are a fast and energy-efficient technology, but need more components in large networks due to poor fan-out/fan-in properties (Mukhanov et al., 1978; Likharev and Semenov, 1991; Katam and Pedram, 2018); furthermore, their weak action potentials are unable to be viewed directly, making diagnostics difficult.
These developing technologies highlight several characteristics that are critical for an artificial neuron: (1) inherent device dynamics that are capable of producing spiking behavior; (2) tunable synapses between neurons to allow for the expansion into larger networks with adjustable connectivity; and (3) low power dissipation, both in the dynamic firing state of the neuron and in the static state. As has been emphasized in prior literature (Sourikopoulos et al., 2017), optimizing static power dissipation is particularly vital to the goal of creating an energy efficient network.
Here, we propose an artificial neuron based on superconducting nanowires operating at cryogenic temperatures. Superconductors are prime candidates for generating low-power spiking behavior due to their inherent non-linearity and negligible static power dissipation. Building on an architecture first implemented in Josephson junctions (Crotty et al., 2010), we rely on the coupling between two shunted nanowires to act analogously to a two-channel model for an action potential. We start by describing the non-linear dynamics of superconducting nanowires, and then present the architecture of the nanowire-based neuron. Using electrothermal circuit simulations, we demonstrate that the device is able to replicate several behaviors of a single neuron. Finally, we present a synapse design and discuss the advantages of the nanowire neuron, including fan-out and an energy figure of merit four orders of magnitude better than that of competing technologies, which are critical in moving toward the parallelism of the human brain.
The Nanowire Neuron Model
Superconducting nanowires possess an inherent non-linearity that serves as the building block of our artificial neuron model. We begin by briefly describing this non-linearity in a single nanowire, and then present the nanowire neuron circuit and its basic operation principles.
The intrinsic non-linearity of superconducting nanowires makes them ideal candidates for the hardware generation of spiking behavior. When a bias current flowing through a superconducting nanowire exceeds a threshold known as the critical current (Ic), superconductivity breaks down and the nanowire becomes resistive, producing a voltage. The nanowire only switches back to the superconducting state once the bias current is reduced below a level called the retrapping current (Ir), and the resistive portion (the “hotspot”) cools down. When the nanowire is placed in parallel with a shunt resistor, this switching process participates in electrothermal feedback with the shunt, producing relaxation oscillations (Toomey et al., 2018).
Relaxation oscillations can be viewed in the context of a simplified action potential. Like the Na+ influx and K+ outflux currents of a neuron, the influx and outflux currents from the nanowire to the shunt resistor are governed by different timescales, τ1 and τ2. As shown in Figure 1A, the rising edge of the output voltage is defined by τ1 = L/(Rs + Rhs), where L is the inductance of the nanowire, Rs is the shunt resistance, and Rhs is the resistance of the nanowire hotspot, usually on the order of ∼1–10 kΩ. Conversely, the outflux current occurs when the nanowire is no longer resistive and the bias current is redirected from the shunt; this reduced resistance results in a slower time constant τ2 = L/Rs which defines the falling edge of the output voltage. For typical nanowires devices, τ1 ∼ 100 ps and τ2 ∼ 1 ns. The two currents are “gated,” as shown by the insets in Figure 1A, by the state of the nanowire—when the state is resistive (producing a voltage), the influx current flows into the shunt, and when it is superconducting, the outflux current flows back to the nanowire.
Figure 1. Relaxation oscillations in superconducting nanowires, which serve as the foundation of the nanowire neuron’s spiking behavior. (A) Simplified model of a relaxation oscillation. The output voltage is defined by two time constants. The rising edge occurs when the superconducting nanowire has switched into the resistive state (Rhs > 0), and the bias current is redirected to the shunt resistor. The falling edge takes place after the nanowire regains superconductivity (Rhs = 0), and the bias current is redirected from the shunt resistor. (B) Example of a superconducting nanowire in a long, meandered design for obtaining a high kinetic inductance. (C) Experimentally measured relaxation oscillations in a long superconducting nanowire shunted by 50 Ω.
The inductance of superconducting nanowires is dominated by an intrinsic material property known as kinetic inductance (Kerman et al., 2006), and is defined per unit length of the structure. As a result, it is possible to tune these time constants by changing the length of the nanowire, with a longer wire leading to a higher inductance and thus a longer timescale. Figure 1B shows a scanning electron micrograph of a typical superconducting nanowire with a meandering geometry designed for maximizing the total device inductance. An example of experimentally observed relaxation oscillations for such a device is displayed in Figure 1C.
The Neuron Model
Although a shunted nanowire on its own produces oscillations analogous to action potentials, as the bias current increases, the output signal eventually accumulates a voltage offset. This effect deviates from true neuron behavior, as the cell must maintain a constant resting potential (approximately −70 mV). To overcome this difference, we have implemented a neuron architecture based on one that was first proposed for Josephson junctions (Crotty et al., 2010), as shown in Figure 2. The circuit consists of two shunted nanowires—the main oscillator and the control oscillator—linked together in a superconducting loop. A bias current Ibias is applied to both oscillators such that they are each biased right below their critical currents, but in opposite directions. To trigger an action potential, a small input current pulse Iin (Figure 2A) is applied and sums with the bias current to exceed Ic of the main oscillator, causing it to switch (Figure 2D). The control does not fire since the input opposes the direction of its bias.
Figure 2. Circuit simulations of the two-nanowire soma, where the two oscillators act analogously to the two ion channels in the simplified neuron model. (A) Input pulse, Iin = 4 μA. (B) Current through the loop inductor. (C) Current through the control nanowire. The control nanowire reduces the amount of counterclockwise current circulating in the loop, allowing the main nanowire to fire again. (D) Current through the main nanowire. (E) Output voltage pulse that is sent to the synapse. For these simulations, the critical current of the control nanowire is Ic,control = 30 μA, the critical current of the main nanowire is Ic,main = 30 μA, and Ibias = 58.6 μA.
Once the main oscillator switches, current is added to the superconducting loop in the counterclockwise direction (Figure 2B), which sums with the bias current to fire the control oscillator (Figure 2C). The control oscillator removes counterclockwise current from the loop, allowing the main oscillator to fire again. Without the presence of the control oscillator, the main oscillator would only be able to fire once, since the counterclockwise current added to the loop would reduce the total current through the nanowire of the main oscillator below its Ic. The voltage from the main oscillator node (Figure 2E) serves as the spiking output that is carried down to the next neuron via the synapse. Unlike the output from the single shunted nanowire, the output of the two-nanowire circuit does not accumulate a bias offset, making it a suitable spiking signal.
In the context of the two-channel neuron model, the main oscillator acts analogously to the Na+ influx current by adding flux to the superconducting loop in the form of a circulating current. The control oscillator acts analogously to the K+ outflux current by reducing the circulating current, resetting the neuron and allowing the main oscillator to fire again. As described in Toomey et al. (2018), the rate at which each oscillator fires depends on the magnitude of the bias current, paralleling the voltage-dependent rate constants of ion gates in the Hodgin-Huxley model (Ermentrout and Terman, 2010).
Single Neuron Characteristics
Neurons display a wide variety of traits unique to certain populations, allowing them to collectively interact to achieve varied and complex tasks. While no single neuron possesses all possible traits, the basic functionality of an artificial neuron can be evaluated by demonstrating some common bio-realistic characteristics. Here we present multiple neuron behaviors that can be achieved with the nanowire neuron, using electrothermal circuit simulations conducted in LTSpice. The simulations implement material-specific characteristics and nanowire hotspot dynamics as described in previous literature (Kerman et al., 2009; Berggren et al., 2018), and have been shown to reliably reproduce experimental data pertaining to nanowire relaxation oscillations (Toomey et al., 2018).
A general characteristic of biological neurons is their inability to fire unless the input signal exceeds a certain threshold. Figure 3A shows the threshold voltage response of the nanowire neuron when the bias current is held constant and the input current is varied. As evident in the plot, the neuron does not begin firing until the input current passes a threshold, defined by when the sum of the bias and input current through the main nanowire exceed its Ic. Above the threshold, the peak voltage of the spike output is essentially constant. However, as emphasized by Izhikevich (2004), biological neurons have a threshold that may be varied by previous activity, such as an inhibiting input that reduces it. Figures 3B,C illustrate this process (“threshold variability”) in the nanowire neuron; an initial subthreshold input pulse (Figure 3B) fails to elicit a spike, while a later input pulse of the same magnitude triggers a spike (Figure 3C) after a smaller negative pulse reduces the firing threshold. It should be noted that when the preceding pulse was of the opposite polarity, no spike was triggered. This behavior is consistent with the expectations of Izhikevich (2004).
Figure 3. Firing threshold of the two-nanowire neuron. (A) Peak output voltage as a function of input current under a constant bias (Ibias = 58.6 μA). The plot illustrates that the neuron does not spike until the input current exceeds 4 μA, at which point it fires with output voltages of the same amplitude. Inset shows the time domain voltage output of the neuron for different input currents. (B) Input to the neuron, leading to a reduction in firing threshold by a preceding negative pulse. (C) Spiking output of the neuron in response to the inputs of (B), demonstrating that the nanowire neuron’s firing threshold is variable. For this simulation, Ibias = 57.62 μA, the positive inputs Iin = 4.6 μA, and the negative input Iin = –4.3 μA.
In addition to exhibiting a firing threshold, the nanowire neuron displays a refractory period, which we define to be the minimum time between two input pulses such that both pulses elicit a spike. Figure 4 illustrates this response. When two pulses are separated enough in time so that the main oscillator is biased close to its critical current when the second input pulse arrives, then the second pulse will cause a spike (Figure 4A). However, if the second pulse arrives before the bias current has fully returned to the main oscillator, then the sum of the second input pulse and the bias will not be sufficient to switch the nanowire and trigger the neuron (Figure 4B). As a result, the refractory period is limited by the time it takes to fully bias the main oscillator again, which is a function of the L/R time constants of the circuit described in Section “Relaxation Oscillations,” as well as the series inductance in the loop. The inductance values stem from the kinetic inductance of the superconducting film, and are therefore dependent on the choice of material, film thickness, and nanowire width and length.
Figure 4. Refractory period of the two-nanowire neuron. (A) Response when there is sufficient time between two inputs to each elicit a separate spike. Parameters: Ibias = 58 μA, Iin = 6 μA, Δt = 4 ns. The pink dashed lines indicate the beginning of the rising edge of each pulse. (B) Response when there is insufficient time between two input pulses, causing the neuron to fire only once. Parameters are the same as in (A), except Δt = 2 ns. For both cases, panel (i) displays the current through the nanowire of the main oscillator, while panel (ii) displays the output voltage of the neuron.
Class I Behavior
Biological neurons differ in their response to varying signal strengths. Whereas Class I neurons have a spiking frequency that increases with increasing input strength, Class II neurons maintain a constant firing rate (Izhikevich, 2004; Crotty et al., 2010). Figure 5 illustrates the spiking behavior of the nanowire neuron at different levels of bias current. Figure 5A shows the time-domain voltage output of the neuron as the bias current is increased, and suggests an increase in spiking frequency. This response is confirmed by observing the voltage output’s frequency spectrum displayed in Figure 5B, which shows a shift in the spiking frequency to higher levels with increasing bias. Consequently, the nanowire neuron has Class I behavior. The modulation of spiking frequency by bias current demonstrates that the frequency of the nanowire neuron output may be used to glean information about its input conditions.
Figure 5. Effect of bias current on spiking frequency. (A) Time domain simulations of the two-nanowire neuron with different bias currents. Iin = 6 μA for all simulations. Traces have been shifted from one another in the y-axis for clarity. (B) Fourier transform of the voltage output for each biasing condition. The shift in peak frequency with bias current indicates that the circuit acts like a Class I neuron.
Axon: Transmission Line Characteristics
After an action potential occurs in a biological neuron, the output signal propagates down the axon as if sent through a delay line (Furber and Temple, 2007). This delay is valuable in that it preserves time domain information, potentially facilitating behaviors that rely on the recognition of specific spatio-temporal patterns (Izhikevich, 2004). Such pattern recognition is often not possible in SNNs with traditional wiring, since signals travel too rapidly for timing information to be maintained (Furber and Temple, 2007). This is not the case for superconducting nanowires that are designed to act as transmission lines. Recent work has shown that the high kinetic inductance of superconducting transmission lines, like those made out of niobium nitride, results in propagation speeds of ∼2% c, where c is the speed of light in vacuum (Zhao et al., 2017b). As illustrated in Figure 6, a simulated nanowire neuron output sent through a superconducting transmission line model (Zhao et al., 2018a) is delayed by ∼100–500 ps, close to the full width of an action potential. In mammalian brains, axonal delays like the cortico-cortical delay (Ferraina et al., 2002) are also on the same timescale as the full width of an action potential [typically a few milliseconds (Bean, 2007)], suggesting that the relative delay in our system with respect to the spike duration is appropriate. If longer delays are needed, the transmission line can simply be made longer.
Figure 6. A superconducting transmission line as an axon. Simulations of a superconducting transmission line show that the spikes can be delayed on the order of ∼0.5 ns, depending on the length of the structure. This could enable the storage of timing information in addition to frequency information. Transmission line parameters: nanowire inductance Ln = 0.3 nH/μm, nanowire capacitance Cn = 0.1 fF/μm, propagation speed v = 1.9% c, transmission line length l = 2.5 mm. Shorter transmission lines on the order of 800 um still had delays of ∼140 ps.
The collective dynamics of a neural network depend on the ability of a neuron to influence the behavior of another downstream neuron via a synapse. Here we introduce an inductive synapse that can be integrated with the nanowire neuron to facilitate downstream control. We start by demonstrating excitatory and inhibitory control, and then present a scheme for tuning the synaptic strength.
The Inductive Synapse
Figure 7A illustrates the circuit schematic of an inductive synapse that may be implemented in the nanowire neuron. Similar to the slow release of neurotransmitters in response to an action potential, the inductive synapse relies on the slow charging of a large inductor in response to the nanowire neuron’s more rapid voltage spikes. The energy stored in the large synapse inductor is then discharged as current into the input port of the target neuron, modulating its behavior.
Figure 7. Nanowire neuron with an inductive synapse. (A) Circuit schematic. The output voltage of the neuron charges the large synapse inductor Lsyn. The inductor discharges current into the input port of the downstream target neuron, modulating its behavior. A series resistor Rseries is in place to reduce backaction into the main neuron. (B) Excitatory downstream control. Parameters: Ibias,main = 59 μA, Rseries = 14 Ω, Rsyn,1 = 40 Ω, Lsyn = 0.265 μH, Rsyn,2 = 40 Ω, Ibias,target = 57.17 μA, Iin = 4.6 μA. (C) Inhibitory downstream control. Parameters Ibias,main = –58.6 μA, Rseries = 24 Ω, Rsyn,1 = 45 Ω, Lsyn = 0.23 μH, Rsyn,2 = 40 Ω, Ibias,target = 57.68 μA, Iin = 4.6 μA. For both cases: Panel (i) displays the output voltage of the main neuron, with the red dashed lines indicating the rising and falling edge of the input signal; Panel (ii) displays the current through the synapse inductor; Panel (iii) displays the output voltage of the downstream target neuron.
The effect of the inductive synapse can be either excitatory or inhibitory based on the sign of the bias current applied to the upstream (main) neuron. These two cases are shown in Figures 7B,C, which displays the voltage outputs of the main and target neurons, as well as the current through the synapse inductor. In the excitatory case (Figure 7B), the upstream neuron is positively biased such that the inductive synapse discharges a positive current, causing an under-biased target neuron to fire. In the inhibitory case (Figure 7C), the upstream neuron is negatively biased, causing a negative current to be sent from the synapse to the target, turning off a target that was biased in the firing state. Consequently, the inductive synapse allows for both types of control between neurons.
Variable Strength Synapse
Although the inductive synapse of Figure 7 can be engineered for both excitatory and inhibitory control of a downstream neuron, a fundamental property of artificial neural networks is the ability to modulate that control by adjusting synaptic strength. One possible scheme for implementing such variability in the inductive synapse is to incorporate superconducting nanowires as a different circuit element: a tunable inductor. A nanowire’s kinetic inductance increases with increasing bias current, reaching an enhancement of 10–20% near Ic (Annunziata et al., 2010). This modulation has been incorporated into the circuit model of the superconducting nanowire used in these simulations (Berggren et al., 2018). By placing a high inductance nanowire with an ideal current source in parallel with the synapse inductor, the overall parallel inductance of the synapse can be modulated, which in turn changes the amount of current sent to the target neuron. Figure 8 shows the simulated results for the case of an inhibitory synapse. When a higher modulating current Imod is applied to the nanowire inductor, the overall parallel inductance increases, reducing the amount of current sent to the target. It is important to note that the polarity of the modulating current is not important, since the change in kinetic inductance depends only on the magnitude of the modulating current in relation to its Ic. For instance, Figure 8B illustrates that the modulation in synaptic current for Imod = 5 μA and Imod = −5 μA is roughly the same. As a result, it is clear that the modulation is due to the change in kinetic inductance, and not simply an injection of current by Imod in the opposing direction.
Figure 8. Modulating the inductive synapse. (A) Circuit schematic of the inductive synapse with a high-inductance nanowire and ideal current source placed in parallel. L1, L2 < < Lnanowire, Lsyn. Rseries, out has been added to further prevent backaction from the target neuron. (B) Simulation of the current through Rseries,out in an inhibitory synapse as a function of different modulation currents. Spikes represent backaction from the firing of the target neuron. (Inset) Enlarged view of the boxed area. Higher modulation currents increase the overall synapse inductance, reducing the amount of current sent to the target. The ±5 μA modulation currents have nearly the same effect, showing that modulation is not polarity-dependent. Parameters: Rseries,in = 25 Ω, Rsyn,1 = 39 Ω, Rsyn,2 = 40 Ω, Rseries,out = 0.1 Ω, Lsyn = 0.45 μH, Lnanowire = 0.275 μH, Ic, nanowire = 6 μA, L1 = L2 = 50 pH, Ibias,target = 57.65 μA, Ibias,main = –59.5 μA.
In conjunction with the plasticity of synapses, the high degree of parallelism in the brain creates a densely connected, adaptable network able to optimize and adjust for different conditions. An example of fan-out in the nanowire neuron is shown in Figure 9. As shown in Figure 9A, a single neuron is connected to four target neurons through four separate tunable synapses. When each of the four modulating currents is set to Imod = 0, the firing of all four targets is inhibited by the main neuron, as illustrated in Figure 9B. To weaken the connection with one of the targets (target #4), Imod,4 is set to 8 μA, turning off the inhibiting action on target #4 but allowing it to remain on the three other target neurons, as shown in Figure 9C. Comparison of the synaptic currents for each of the four targets shows that the synaptic current for target #4 is reduced as a result of the modulated inductance. This modulation illustrates that the nanowire neuron is able to be used in a parallel network where the strength of individual synaptic connections can be adapted.
Figure 9. Fan-out of nanowire neuron with a tunable inductive synapse. (A) Simplified circuit model of the fan-out circuit. (B) Simulation when Imod = 0 μA for all four target neurons. (i) Output of the main upstream neuron. (ii) Current through each of the four series resistors Rseries, out to each of the target neurons. (iii) Output voltage of the four target neurons, shifted on the y-axis for clarity. (C) Simulation when Imod = 0 μA for targets #1–3 and Imod = 8 μA for target #4. (i) Output of the main upstream neuron. (ii) Current through each of the four series resistors Rseries, out to each of the target neurons. The current through Rseries,out for target #4 is less than that of the other targets, showing that it has been modulated. (iii) Output voltage of the four target neurons, shifted on the y-axis for clarity. The first three targets have been inhibited, while the inhibitory action on target #4 has been turned off. Parameters: Rseries,in = 7 Ω, Rsyn,1 = Rsyn,2 = 300 Ω, Lsyn = 0.4 μH, Rseries,out = 1 Ω, Lnanowire = 0.275 μH, Ic, nanowire = 6 μA, Ibias,main = –59.5 μA, Ibias,target = 57.65 μA.
Fan-out may be one area where nanowires will be an improvement over Josephson junctions (Crotty et al., 2010), another superconducting technology with promise for artificial neurons. Both nanowires and Josephson junctions have quantized flux outputs (flux here being defined as the time-integral of the voltage). However, Josephson devices have outputs of only a single flux-quantum, while nanowires typically have outputs with many more flux quanta (e.g., 70 flux quanta for the device in Figure 2). For instance, typical niobium nitride nanowires with critical currents in the range of 100 μA and film thicknesses around 10 nm have flux outputs ranging from 50 to 100 flux quanta. In pushing the fan-out and fan-in to larger systems, one expects to be eventually limited by parasitic inductances and thermal noises. In such a case, the more substantial signal of the nanowire neuron would permit a larger fan-out and fan-in, leading to a higher degree of parallelism. In addition to the fan-out, the nanowire voltage signals are long enough and large enough to be digitized directly on an oscilloscope, in contrast to their Josephson junction counterparts, allowing for more direct analysis and readout. Finally, there are a suite of three-terminal, power-control devices made from nanowires (McCaughan and Berggren, 2014), which could be fabricated alongside the neurons and used to boost signals and match impedances.
Benchmarking Synaptic Energy and Speed
The energy dissipation of both the neuron and the synapse can be calculated using LTSpice by taking the time integral of the current-voltage product of each circuit element. Performing this analysis, we find that the nanowire neuron has an energy of about 0.05 fJ for each action potential, while the synapse is about an order of magnitude less at around 0.005 fJ. In large systems, it will be the synapses that will dominate; typically if there are O(N) neurons there will be O(N2) synapses. Hence, even though the neuron dissipates more energy, it will be the synapses that will dominate the power consumption of a large network.
In a SNN, the energy dissipated increases with the speed of the system; spiking twice as often dissipates twice the energy, assuming the energy per spike stays constant. The appropriate figure of merit to compare different technologies is then to take the ratio of speed and power. IBM (Merolla et al., 2014) introduced the figure of merit of synaptic operations per second per watt (SOPS/W). We include a constant factor of about 400 W/W for the nanowire neuron to account for the cryogenic cooling costs. Table 1 compares both the energy per spike and the SOPS/W for the nanowire neuron, the human brain, and two CMOS technologies. We acknowledge that the estimate for the nanowire neuron is a projection from a calculation of a single component, whereas the other entries in the table have actually been measured on large systems. However, given that superconducting platforms have no dissipation in their interconnects in their DC state, we believe that it is reasonable to project in such a way. We have also assumed that the constant applied bias current will not significantly affect the overall power consumption, based on previously proposed superconducting architectures that use a current dividing network made of inductors rather than resistors to eliminate the majority of dissipation from static biasing, as has been suggested for single flux quantum electronics (Mukhanov, 2011). In doing so, it is apparent that the nanowire neuron can be a highly competitive technology from a power and speed perspective, and that operation at cryogenic temperatures does not pose a serious disadvantage to its overall performance.
Table 1. Energy and figure of merit values for different artificial neural networks in comparison to those of the human brain.
Density is also a relevant parameter for building large-scale neural networks. At present, we offer a conservative estimate that an initial proof-of-concept device would have a cell size of at most 50 μm × 50 μm. We believe this area would likely be made smaller by refining the circuit parameters, the cell layout, and choice of materials. For instance, a superconducting film with a higher kinetic inductance would lead to a smaller device area. However, we imagine this technology initially being used in a centralized cloud-based architecture, where the neuron’s low power consumption relative to that of competing platforms would compensate for the lower device density. This type of tradeoff has been justified in past literature, such as the adiabatic quantum flux parametron which similarly sacrifices device area for low-power logic (Takeuchi et al., 2013). Thus, while the anticipated device density of the nanowire neuron can be improved, its speed and power dissipation offer significant advantages that could compensate for its estimated cell size.
Discussion on Possible Synapse Alternatives
Although the fan-out achieved here is far from the level of parallelism in the human brain (where each neuron connects to 1000s of neighbors), it suggests that nanowires may serve a unique purpose in the development of future superconducting neural networks, which have thus far struggled to support fan-out. Given that nanowires can interface with both CMOS and Josephson junction circuits (Zhao et al., 2017a), it may be possible for nanowire neurons to serve as intermediary devices in a network with both platforms. Indeed, a recently proposed neural network with hybrid technologies employed superconducting nanowires as photon detectors, relying instead on optical signals for facilitating high fan-out (Shainline et al., 2018; Shainline, 2019). Although our work uses nanowires solely as electrical components, they can easily be biased to act as photodetectors (Goltsman et al., 2001; Marsili et al., 2011) as well, illustrating that the two different architectures would be compatible for integration. These two schemes thus illustrate the diverse ways in which superconducting nanowires can be used in neural networks, suggesting that a combination of the two technologies may be possible.
Furthermore, while the inductive synapse proposed in this work relies on an external modulating bias current, it may be possible to use a superconducting memory cell to store the value of this modulating current, or replace it directly in the circuit. Memory cells based on superconducting nanowire loops have been shown to reliably store states in the form of a trapped circulating current (Murphy et al., 2017; McCaughan et al., 2018; Zhao et al., 2018b), with recent work demonstrating that one can program and store different amounts of current in the loop in discrete quantities (Toomey et al., 2019). Incorporating these programmable memory cells into the modulating elements of the inductive synapse could enable storage of a varying synaptic strength in each synapse, allowing for more complex applications. It may also be possible for nanowire neurons to connect through alternative synapse designs, such as one based on inductive coupling. Further analysis is needed to verify the possibility of fan-out in such a scheme.
By taking advantage of intrinsic non-linearities in superconducting nanowires, we have designed a platform for a low-power artificial neuron. In this platform, the coupling of two nanowire-based oscillators acts analogously to the two ion channels in a simplified neuron model, producing an output voltage spike that serves as the information-carrying token in these circuits. Using electrothermal circuit simulations, we have shown that the nanowire neuron is able to reproduce universal biological neuron characteristics, such as a firing threshold, as well as unique characteristics distinct to certain neural classes, such as parabolic bursting (see Supplementary Material). Furthermore, we suggested that a nanowire transmission line with a propagation speed of ∼2% c may be used as an axon delay line, potentially allowing spatio-temporal information to be accessed. These collective behaviors may enable the nanowire neuron to be used in a rich variety of operations.
In addition to harnessing the non-linearity of the nanowire’s switching dynamics, we relied on the non-linearity of the nanowire’s kinetic inductance in order to develop a variable inductive synapse. We demonstrated that the total parallel inductance of the synapse can be tuned with a modulating bias current, thereby changing the strength of the signal sent to a downstream target neuron. This scheme proved to be capable of fan-out, as demonstrated by a single neuron inhibiting the firing of four target neurons. An energy analysis of this circuit in comparison to other spiking networks illustrated that the nanowire neuron has competitive performance in the dynamic firing state and a figure of merit four orders of magnitude better than certain alternative platforms, while the static state benefits from the lack of power dissipation by the superconducting elements. Although experimental realization of these results is a subject of future work, the analysis performed here suggests that the nanowire neuron is a promising candidate for the advancement of low-power artificial neural networks.
Looking forward, networks of superconducting nanowires could be the basis for powerful new computer hardware. Analog blocks of highly connected neurons could be digitally linked to achieve networks with either scale-free or small-worlds connectivity. With superconducting interconnects, entire chips could be wired together with no cost in heat dissipation. The result would be a large-scale neuromorphic processor which could be trained as a SNN to perform tasks like pattern recognition or used to simulate the spiking dynamics of a large, biologically-realistic network. The combination of speed, low power dissipation, and biological realism with only a few components suggests that nanowires could outperform or complement other existing and developing neuromorphic hardware technologies.
The nanowire LTSpice model used for this study can be found on GitHub at https://github.com/karlberggren/snspd-spice. All circuit schematic and data files, including the Matlab code used to generate the figures, are available from the authors upon request.
ET performed the simulations with input from KS and KB. All authors contributed to the design of the circuits and the writing of the manuscript.
ET was supported by the National Science Foundation Graduate Research Fellowship Program (NSF GRFP) (Grant No. 1122374). The authors would like to acknowledge the generous support of the Bose Foundation.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The authors thank Murat Onen, Brenden Butters, and Jason Meyers for scientific discussions, Akshay Agarwal and Dr. Donnie Keithley for edits, and the entire Quantum Nanostructures and Nanofabrication group.
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2019.00933/full#supplementary-material
Annunziata, A. J., Santavicca, D. F., Frunzio, L., Catelani, G., Rooks, M. J., Frydman, A., et al. (2010). Tunable superconducting nanoinductors. Nanotechnology 21:445202. doi: 10.1088/0957-4484/21/44/445202
Berggren, K. K., Zhao, Q. Y., Abebe, N., Chen, M., Ravindran, P., McCaughan, A., et al. (2018). A superconducting nanowire can be modeled by using SPICE. Supercond. Sci. Technol. 31:055010. doi: 10.1088/1361-6668/aab149
Ferraina, S., Paré, M., and Wurtz, R. H. (2002). Comparison of cortico-cortical and cortico-collicular signals for the generation of saccadic eye movements. J. Neurophysiol. 87, 845–858. doi: 10.1152/jn.00317.2001
Goltsman, G. N., Okunev, O., Chulkova, G., Lipatov, A., Semenov, A., Smirnov, K., et al. (2001). Picosecond superconducting single-photon optical detector. Appl. Phys. Lett. 79, 705–707. doi: 10.1063/1.1388868
Kerman, A. J., Dauler, E. A., Keicher, W. E., Yang, J. K. W., Berggren, K. K., Gol’tsman, G., et al. (2006). Kinetic-inductance-limited reset time of superconducting nanowire photon counters. Appl. Phys. Lett. 88:111116. doi: 10.1063/1.2183810
Marsili, F., Najaf, F., Dauler, E., Bellei, F., Hu, X., Csete, M., et al. (2011). Single-photon detectors based on ultranarrow superconducting nanowires. Nano Lett. 11, 2048–2053. doi: 10.1021/nl2005143
McCaughan, A. N., Toomey, E. A., Schneider, M., Berggren, K. K., and Nam, S. W. (2018). A kinetic-inductance-based superconducting memory element with shunting and sub-nanosecond write times. Supercond. Sci. Technol. 32:015005. doi: 10.1088/1361-6668/aae50d
Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, J., et al. (2014). A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673. doi: 10.1126/science.1254642
Murphy, A., Averin, D. V., and Bezryadin, A. (2017). Nanoscale superconducting memory based on the kinetic inductance of asymmetric nanowire loops. New J. Phys. 19:063015. doi: 10.1088/1367-2630/aa7331
Querlioz, D., Bichler, O., and Gamrat, C. (2011). “Simulation of a memristor-based spiking neural network immune to device variations,” in Proceedings of the 2011 International Joint Conference on Neural Networks, San Jose, CA.
Schneider, M. L., Donnelly, C. A., Russek, S. E., Baek, B., Pufall, M. R., Hopkins, P. F., et al. (2018). Ultralow power artificial synapses using nanotextured magnetic Josephson junctions. Sci. Adv. 4:e1701329. doi: 10.1126/sciadv.1701329
Shainline, J. M., Buckley, S. M., McCaughan, A. N., Chiles, J., Mirin, R. P., Nam, S. W., et al. (2018). Circuit designs for superconducting optoelectronic loop neurons. J. Appl. Phys. 124:152130. doi: 10.1063/1.5038031
Sourikopoulos, I., Hedayat, S., Loyez, C., Danneville, F., Hoel, V., Mercier, E., et al. (2017). A 4-fJ/Spike artificial neuron in 65 nm CMOS technology. Front. Neurosci. 11:123. doi: 10.3389/fnins.2017.00123
Takeuchi, N., Ozawa, D., Yamanashi, Y., and Yoshikawa, N. (2013). An adiabatic quantum flux parametron as an ultra-low-power logic device. Supercond. Sci. Technol 26:035010. doi: 10.1088/0953-2048/26/3/035010
Toomey, E., Onen, M., Colangelo, M., Butters, B. A., McCaughan, A. N., and Berggren, K. K. (2019). Bridging the gap between nanowires and josephson junctions: a superconducting device based on controlled fluxon transfer. Phys. Rev. Appl. 11:034006.
Zhao, Q.-Y., McCaughan, A. N., Dane, A. E., Berggren, K. K., and Ortlepp, T. (2017a). A nanocryotron comparator can connect single-flux-quantum circuits to conventional electronics. Supercond. Sci. Technol. 30:044002. doi: 10.1088/1361-6668/aa5f33
Zhao, Q.-Y., Santavicca, D. F., Zhu, D., Noble, B., and Berggren, K. K. (2018a). A distributed electrical model for superconducting nanowire single photon detectors. Appl. Phys. Lett. 113:082601. doi: 10.1063/1.5040150
Zhao, Q.-Y., Toomey, E. A., Butters, B. A., McCaughan, A. N., Dane, A. E., Nam, S. W., et al. (2018b). A compact superconducting nanowire memory element operated by nanowire cryotrons. Supercond. Sci. Technol. 31:035009. doi: 10.1088/1361-6668/aaa820
Keywords: artificial neuron, superconductor, nanowire, spiking neural network, artificial synapse
Citation: Toomey E, Segall K and Berggren KK (2019) Design of a Power Efficient Artificial Neuron Using Superconducting Nanowires. Front. Neurosci. 13:933. doi: 10.3389/fnins.2019.00933
Received: 26 June 2019; Accepted: 20 August 2019;
Published: 04 September 2019.
Edited by:Yoeri van de Burgt, Eindhoven University of Technology, Netherlands
Reviewed by:Jessamyn Fairfield, National University of Ireland Galway, Ireland
Junxiu Liu, Ulster University, United Kingdom
Armantas Melianas, Stanford University, United States
Copyright © 2019 Toomey, Segall and Berggren. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.