High-performance computing has historically developed around the Boolean computing paradigm, executed on silicon (Si) complementary metal oxide semiconductor (CMOS) hardware. In fact, software has for decades been developed around the CMOS fabric that has singularly dictated our choice of materials, devices, circuits, and architecture–leading to the dominant processor design paradigm: von Neumann architecture that separates memory and processing units. Over the last decade, however, Moore’s law for hardware scaling has significantly slowed down, primarily due to the prohibitive energy cost of computing and an increasingly steep memory wall. At the same time, software development has significantly evolved around “Big Data” paradigm, with machine learning and artificial intelligence (AI) dominating the roost. Additionally, the push towards the internet of things (IoT) edge devices has prompted an intensive search for energy-efficient and compact hardware systems for on-chip data processing (Big data, 2018).

One such direction is neuromorphic computing, which uses the concept of mimicking a human brain architecture to design circuits and systems that can perform highly energy-efficient computations (Mead, 1990; Schuman et al., 2017; Marković et al., 2020; Christensen et al., 2022; Kireev et al., 2022). A human brain is primarily composed of two functional elemental units - synapses and neurons. Neurons are interconnected through synapses with different connection strengths (commonly known as synaptic weights), which provide the learning and memory capabilities of the brain. A neuron receives synaptic inputs from other neurons, generates output in the form of action potentials, and distributes the output to the subsequent neurons. A human brain has ∼ 1 0 11 neurons and ∼ 1 0 15 synapses and consumes ∼ 1 − 10 f J per synaptic event (Kandel et al., 2000; Squire et al., 2012; Upadhyay et al., 2016).

To emulate the organization and functionality of a human brain, there are many proposals for physical neuromorphic computing systems using memristors (Yao et al., 2020; Duan et al., 2020; Moon et al., 2019), spintronics (Grollier et al., 2020; Locatelli et al., 2014; Lv et al., 2022), charge-density-wave (CDW) devices (Liu et al., 2021), photonics (Shastri et al., 2021; Shainline et al., 2017), etc. In recent years, there has been significant progress in the development of physical neuromorphic hardware, both in academia and industry. The hierarchy of neuromorphic hardware implementation spans from the system level to the device level and all the way down to the level of the material. At the system level, various large-scale neuromorphic computers utilize different approaches - for instance, IBM’s TrueNorth (Merolla et al., 2014), Intel’s Loihi (Davies et al., 2018), SpiNNaker (Furber et al., 2014), BrainScaleS (Schemmel et al., 2010), Tianjic chip (Pei et al., 2019), Neurogrid (Benjamin et al., 2014), etc. They support a broad class of problems ranging from complex to more general computations. At the device level, the most commonly used component is the memristor which can be utilized in synapse and neuron implementations (Jo et al., 2010; Serb et al., 2020; Innocenti et al., 2021; Mehonic and Kenyon, 2016). Memristor crossbars are frequently used to represent synapses in neuromorphic systems (Adam et al., 2016; Hu et al., 2014). Memristor can also provide stochasticity in the neuron model (Suri et al., 2015). Another emerging class of devices for neuromorphic computing is spintronics devices (Grollier et al., 2020). Spintronics devices can be implemented with low energy and high density and are compatible with existing CMOS technology (Sengupta et al., 2016a). The spintronics devices utilized in neuromorphic computing include spin-torque devices (Torrejon et al., 2017; Roy et al., 2014; Sengupta et al., 2016b), magnetic domain walls (Siddiqui et al., 2020; Leonard et al., 2022; Brigner et al., 2022), and skyrmions (Jadaun et al., 2022; Song et al., 2020). Optical or photonics devices are also implemented for neurons and synapses in recent years (Shastri et al., 2021; Romeira et al., 2016; Guo et al., 2021). The field is very new and many novel forms of neuron and synaptic devices can be designed to match the mathematical model of neural networks (NNs). Physical neuromorphic computing can implement these functionalities directly in their physical characteristics (I-I, V-V, I-V), which results in highly compact devices that are well-suited for scalable and energy-efficient neuromorphic systems (Camsari et al., 2017a; Camsari et al., 2017b; Ganguly et al., 2021; Yang et al., 2013). This is critical as current NN-based computing is highly centralized (resident-on and accessed-via cloud) and is energy inefficient because the underlying volatile, often von Neumann, digital Boolean-based system design unit has to emulate inherently analog, mostly non-volatile distributed computing model of neural systems, even if at a simple abstraction level (Merolla et al., 2014). Recent advances in custom design such as FPGAs (Wang et al., 2018) and more experimental Si FPNAs (Farquhar et al., 2006) have demonstrated that a new form of device design rather than emulation is the way to go, and physical neuromorphic computing based on emerging technology can go a long way to achieve this (Rajendran and Alibart, 2016).

There is an increased use of noise-as-a-feature rather than a nuisance in NN models (Faisal et al., 2008; Baldassi et al., 2018; Goldberger and Ben-Reuven, 2017), and physical neuromorphic computing can provide natural stochasticity, with various noise colors depending on the device physics (Vincent et al., 2015; Brown et al., 2019). Some prominent areas where stochasticity and noise have been used include training generalizability (Jim et al., 1996), stochastic sampling (Cook, 1986), and recently proposed and coming into prominence, diffusion-based generative models (Huang et al., 2021). In all these models, noise plays a fundamental role, i.e., these algorithms do not work without inherent noise.

It is therefore critical to study and analyze the kinds of devices that will be useful to implement physical neuromorphic computing. We understand from neurobiology that there is a large degree of neuron design customization that has developed through evolution to obtain high task-based performance. Similarly, a variety of mathematical models of neurons have been designed in NN literature as well (Schuman et al., 2017; Burkitt, 2006; Ganguly et al., 2021). It is quite likely that the area of physical neuromorphics will use a variety of device designs rather than the uniformity of NAND gate-based design commonly seen in Boolean-based design, to achieve the true benefits of energy efficiency and scalability brought forth by this paradigm of system design.

In this work, we study a subset of this wide variety of neuron designs that are well-represented and easily available from many proposed physical neuromorphic platforms to understand and analyze their task specialization. In particular, we analyze analog and binary neuron models, including stochasticity in the model, for analog temporal inferencing tasks, and evaluate and compare their performances. We numerically estimate the performance metric normalized means squared error (NMSE), discuss the effect of stochasticity on prediction accuracy vs. robustness, and show the hardware implementability of the models. Furthermore, we estimate the memory capacity for different neuron models. Our results suggest that analog stochastic neurons perform better for analog temporal inferencing tasks both in terms of prediction accuracy and hardware implementability. Additionally, analog neurons show larger memory capacity. Our findings may provide a potential path forward toward efficient neuromorphic computing.

An essential function of a neuron in a NN is processing the weighted synaptic inputs and generating an output response. A single biological neuron itself is a complex dynamical system (Bick et al., 2020). Proposed artificial neurons in most implementations of NNs (either software or hardware) are significantly simpler unless they specifically attempt to mimic the biological neuron (Harmon, 1959; Schuman et al., 2017; 2022). As such their mathematical representations are cheaper and a significant amount of computational capabilities derive from the network itself. However, a NN is an interplay of the neurons, the synapses, and the network structure itself, and therefore the neuron model itself may provide certain capabilities that can help make a more efficient NN, in the context of the application specialization (Abiodun et al., 2018).

The set of behavior over which such neurons can be classified and analyzed is vast and may include spiking vs. non-spiking behavior with associated data representation, deterministic vs. stochastic output response function, discrete (or binary) vs. continuous (or analog) output response function, the particular mathematical model of the output response function itself (e.g., sigmoid, tanh, ReLU), presence or absence of memory states with a neuron, etc (Goodfellow et al., 2016; Davidson and Furber, 2021; Barna and Kaski, 1990). In the software NN world, specialization of certain neural models and connectivity are well appreciated, as an example sparse vs. dense vs. convolutional layers, or the use of ReLU neurons in the hidden layers vs. sigmoidal, softmax layers at outputs employed in many computer vision tasks (Szandała, 2020; Zhang and Woodland, 2015; Oostwal et al., 2021). Figure 1A schematically shows the output characteristics of different types of widely used neuron models.

In this work, we have focused on two particular behaviors of neural models that we believe can capture a significant application space, particularly in the domain of lightweight real-time signal processing tasks, and are readily built from emerging materials technology. We specifically look at binary vs. analog and deterministic vs. stochastic neuron output response functions (purple-colored bold font labels in Figure 1A). We also use them in a reservoir computing (RC)-like context for signal processing tasks for our analysis. Reservoir computing uses the dynamics of a recurrently connected network of neurons to project an input (spatio-)temporal signal onto a high dimensional phase space, which forms the basis of inference, typically via a shallow 1-layer linear transform or a multi-layer feedforward network (Tanaka et al., 2019; Triefenbach et al., 2010; Jalalvand et al., 2015; Ganguly et al., 2018; Moon et al., 2019). A schematic of a reservoir is shown in Figure 1B where the neurons are connected with each other bidirectionally with random weights. Multiple reservoirs may be connected hierarchically for more complex deep RC architecture. RC may be considered as a machine learning analog of an extended Kalman filter where the state space and the observation models are learned and not designed a priori (Tanaka et al., 2019).

Our choice of evaluating these specific behavior differences on an RC-based NN reflects the prominent use-case that is made out for many emerging nano-materials technology-based neuron and synaptic devices, viz. energy-efficient learning, and inference at the edge. These tasks often end up involving temporal or spatio-temporal data processing to extract relevant and actionable information, some examples being anomaly detection (Kato et al., 2022), feature tracking (Abreu Araujo et al., 2020), optimal control (Engedy and Horváth, 2012), and event prediction (Pyragas and Pyragas, 2020), all of which are well-suited for an RC-based NN. Therefore this testbench forms a great intersection for our analysis.

It should be noted that we do not include spiking neurons in this particular analysis. Spiking neurons have significantly different data encoding (level vs. rate or inter-spike interval encoding) and learning mechanisms (back-propagation or regression vs. spike-time dependent plasticity) that it is hard to disentangle the neuron model itself from demonstrated tasks, therefore we leave such a contrasting analysis of spiking neuron devices with non-spiking variants for a future study.

The neurons are modeled in the following way: y = f N ∑ w T x + r N (1)

Here the symbols have the usual meaning, i.e., y is the output activation of the neuron, f _N is the activation function, which is a sigmoidal or hyperbolic tangent for most non-spiking hardware neurons, and r _N is a random sample drawn from a random uniform distribution to represent stochasticity. It is possible to use a ReLU-like activation function or some other distribution for sampling stochasticity, particularly if the hardware neuron shows colored noise behavior, we do not particularize for such details and keep the analysis confined to the most common hardware neuron variants. Therefore, in our analysis, the r _N term is weighed down by an arbitrary factor to mimic the degree of stochasticity displayed by the neuron, and the f _N is either a continuous tanh() for analog neuron or a sgn(tanh()) for a binary neuron (sgn() being the signum function).

As discussed previously, the neuron models are analyzed in the context of a reservoir computer, specifically an echo-state network (ESN). An ESN is composed of a collection of recurrently connected neurons, with randomly distributed weights of the interconnects within this collection (Lukoševičius, 2012; Li et al., 2012). This forms the “reservoir”, which is activated by an incoming signal, and whose output is read by an output layer trained via linear regression.

We employ different neuron models in this work, such as analog and binary neurons (with and without stochasticity in the model), which makes a total of four models at our disposal, namely, analog neuron (AN), analog stochastic neuron (ASN), binary neuron (BN), and binary stochastic neuron (BSN). The dynamical equations of the reservoirs built using different neuron models are described as follows (Ganguly et al., 2021): AN : x t + 1 = 1 − a * x t + a * tanh z t + 1 ASN : x t + 1 = 1 − a * x t + a * tanh z t + 1 + b * r N t BN : x t + 1 = 1 − a * x t + sgn a * tanh z t + 1 BSN : x t + 1 = 1 − a * x t + sgn a * tanh z t + 1 + b * r N t (2) where z[t + 1] = W ⁱⁿ u[t + 1] + W ^s x[t]. Here, u is the input vector, x[t] represents the reservoir state vector at the time t, a is the reservoir leaking rate (assumed to be the constant for all the neurons), b is the neuron noise scaling parameter to include stochasticity in the neuron model, r _N is a uniform random distribution, and W ⁱⁿ and W ^s are the random weight matrices of input-reservoir and reservoir-reservoir connections, respectively. We use the same leaking rate across all models to ensure a fair comparison among the neuron models on an equal footing. It can be challenging to compare models that have different parameters as it can introduce biases. One of the unique features of reservoir computing is having random weight matrices (Tanaka et al., 2019) and we consider five different network topologies by creating five sets of W ^s using random “seed” for various reservoir sizes, which makes our analysis unbiased to any particular network topology. The W ^s elements are normalized using the spectral radius. We perform 1,000 simulations within each network topology making the total sample size 5,000 for every reservoir size within each neuron model. The output vector y is obtained as: y = W out x (3) where W ^out represents the reservoir-output weight matrix. We consider two different types of training methods, i.e., “offline” and “online” training. In the case of “offline” training, we extract the output weight matrix, W ^out once at the end of the training cycle and use that static W ^out for the testing cycle. In contrast, for “online” training, W ^out is periodically updated throughout the testing cycle. The entire testing cycle is divided into 40 segments. The first segment uses the W ^out extracted from the initial training cycle. We calculate a new W ^out after the first segment of the testing cycle. Then, we update the W ^out such that the elements are composed of 90% from the older version and 10% from the new one. The updated W ^out is used for the second segment and the procedure keeps going on throughout the testing cycle. This stabilizes the learning at the cost of higher error rates as the learning evolution slowly evolves to a new configuration. This is akin to the successive over-relaxation methods used in many self-consistent numerical algorithms for improved convergence.

In summary, we studied different neuron models for the analog signal inferencing (time-series prediction) task in the context of reservoir computing and evaluate their performances for various input functions. We show that the performance metrics are better for ASN than BSN for both clean and distorted input signals. We find that the increasing degree of stochasticity makes the models more robust, however, decreases the prediction accuracy. This introduces a trade-off between accuracy and robustness depending on the application requirements and specifications. Furthermore, the ASN model turns out to be the suitable one for hardware implementation, which attributes to the smallest dynamics range of the learned synaptic weights, although other aspects, i.e., energy requirement, scalability, and reconfigurability need to be assessed. Additionally, we estimate the linear memory capacity for different neuron models, which suggests that analog neurons have a higher ability to reconstruct the past input signal from the present reservoir state. These findings may provide critical insights for choosing suitable neuron models for real-time signal-processing tasks and pave the way toward building energy-efficient neuromorphic computing platforms.