An SNN employs a spiking neuron as the basic computational unit with input and output in the form of spikes, maintaining an internal membrane potential over time. In this paper, we adopt the Spike Response Model (SRM) which phenomenologically describes the dynamic response of biological neurons.

Consider an input spike, sjl-1(t)=δ(t-tj(l-1)). Here tj(l-1) denotes a firing time of pre-synaptic neuron j in layer l − 1 and δ the spike function. In the SRM model, the incoming spike sjl-1(t) is converted into spike response signals by convolving with the spike response kernel ϵ(t) and is then scaled by the synaptic weight to generate the Post Synaptic Potential (PSP). Likewise, the refractory period can be represented as (ν*sjl)(t) which describes the characteristic recovery time needed before the neuron regains its capacity to fire again after having fired at time t. The neuron's membrane potential, is the sum of all PSPs and refractory response (1)uil(t)=∑jWijl-1(ϵ*sjl-1)(t)+(ν*sil)(t) where uil(t) is the membrane potential of neuron i and Wijl-1 is the synaptic weight from neuron j to neuron i.

A firing output is generated wherever u_i(t) crosses the predefined firing threshold θ_u. This generation process can be formulated by a Heaviside step function Θ as follows (2)sil(t)=Θ(uil(t)-θu).

As shown in Figure 1, a VAD is added to the output of each spiking neuron in layer l. Let N be the number of neurons at layer l, thus, the set of spike trains s^l(t) can be represented as follows (3)sl(t)={s1l(t),...,sNl(t)} The forward pass of the delay module can be described as (4)sdl(t)=δ(t-dl^)*sl(t) Where d^l is the set of learnable delays {d^1,d^2,..,d^N} in layer l, and sdl(t) is the spike trains output by the delay module. From the system point of view, limiting the axonal delay of each neuron to a reasonable range can speed up the training convergence. Thus, we clip the delay to the specified range during training and round down after each backpropagation. (5)d^l=Min(Max(0,round(d^)),θd) Here, the θ_d refers to the upper bound of the time delay of the spiking neuron.

The structure of the local skip-connection within a given layer is depicted in Figure 2. In mapping from input spikes to output spikes, The SRM utilizes a refractory kernel to characterize the refractory mechanism, represented by the equation ν(t)=-αrθutτrexp(1-tτr)Θ(t). One challenge that persists is identifying the ideal refractory scale α_r for specific tasks. If the refractory scale is too small, its effect is diminished, while an overly large refractory scale risks information loss at certain time junctures. To address this, our study introduces the concept of a local skip-connection. This design compensates for information lost during the reset mechanism in a dynamic fashion. Our results show that this connection can operate effectively using the same refractory scale, offering a solution to the intricate task of selecting an optimal refractory scale for various tasks. The output membrane potential of the local skip-connection can be formulated as (6)ûil(t)=∑jVijl(ϵ*sd,jl)(t)+(ν*ŝil)(t)

Vijl is the locally connected synaptic weight from neuron j to neuron i at the same layer. Unlike a skip connection, the local skip-connection adds an extra layer of processing to the output spikes generated in layer l. It then directs these locally processed output spikes, denoted as ŝ^l with the same index as the original output spikes sdl, to follow the same axon line within layer l. As a result, both the local spike trains ŝ^l and the original output spikes sdl utilize the same weights Wijl and are channeled to the succeeding layer. This can be equivalently expressed as sl=sdl+ŝl.

The loss of an SNN compares the output spikes with the ground truth. However, in classification tasks, decisions are typically made based on the spike due to the absence of precise timing. Considering the spike rate over the time interval T, the loss function L can be formulated as follows: (7)L=12(∫0Ts~(τ)dτ-∫0Tsnl(τ)dτ)2 Here, L measures the disparity between the target spike train s~(t) and output spike train s^nl(t) at the last layer n_l across the simulation time T. Given the lack of precise spike timing in our tasks, we measure the output spikes through the integration of snl(t) over T. For different task scenarios, the target firing rate is set as ∫0Ts~(τ)dτ.

To further exploit temporal information in classification, an auxiliary loss termed the suppressed loss L_Mem is introduced: (8)LMem(t)=12·(snl(t)·Mask·(unl(t-Δt)-uθ))2 This loss function is designed to reduce the firing probability of incorrect neurons right when they activate. Compared to previous lateral inhibition methods using learnable or fixed kernels, this loss function achieves a winner-takes-all effect by acting as a regularizer. Importantly, this loss is only applied to false neurons. Here, the spike train snl and membrane potential unl are functions of time. Moreover, unl(t-Δt) refers to the membrane potential right before a spike occurs. When a neuron is activated, indicated by snl(t)=1, its potential is referred to as unl(t-Δt). This value is then subtracted from a predetermined membrane potential u_θ, controlled by the suppressing factor λ_u and defined as u_θ = λ_uθ_u. Lastly, to ensure that the suppressed membrane potential loss is limited only to undesired (or false) neurons, a mask Mask ∈ ℝ^C is employed, where C is the number of target neurons: (9)Mask={0True Class1False Classes

The surrogate gradient algorithm in combination with the Backpropagation-Through-Time (BPTT) (Werbos, 1990) in SNN has shown excellent performance on temporal pattern recognition tasks.

In this work, we discretise the temporal dimension with the sampling time T_s such that t = nT_s where n denotes the time step of the simulation. We also define (N_s + 1)T_s as the total observation time. For the Heaviside step function, we adapt the SLayer function (Shrestha and Orchard, 2018) to formulate the proxy gradient, which is defined as (10)fs ′^=τscaleexp(-|u(t)-ϑ|/τϑ) Here, τ_scale and τ_ϑ are two parameters that control the sharpness of the surrogate gradient. Similarly, the gradient of the axonal delay is given by (11)∇d^lE=Ts∑n=0Ns∂L[n]∂d^l Using the chain rule and noting that the loss at time-step n depends on all previous timesteps, we get (12)∇d^lE=Ts∑n=0Ns∑m=0n∂sdl[m]∂dl∂L[n]∂sdl[m]=Ts∑n=0Ns∑m=0nsdl[m]-sdl[m-1]Ts∂L[n]∂sdl[m] Here, the finite difference approximation sdl[m]-sdl[m-1]Ts is used to numerically estimate the gradient term ∂sdl[m]∂dl. As part of the backpropagation process, the gradient of delay is propagated backward, and then the delay value is subsequently updated. Similarly, we also formulate the gradient term of the suppressed loss. (13)  ∂LMem∂unl=snl·Mask·(unl-uθ) As shown in Figure 2, beginning from the input layer, the spike trains compute forward and the error propagates backward.

The experiments are conducted using PyTorch as a framework, and all reported results are obtained on 1 NVIDIA Titan XP GPU. Each network and proposed architecture is trained with the Adam optimizer (Kingma and Ba, 2014) and has the same training cycle. The simulation time step T_s is 1 ms, and the firing threshold θ_u is set at 10 mV. The chosen response kernel is ϵ(t)=tτsexp(1-tτs)Θ(t), and the refractory kernel is ν(t)=-αrθutτrexp(1-tτr)Θ(t). The time constant of the response kernel τ_s and refractory kernel τ_r is set to 5 for NTIDIDIGITS and 1 for SHD datasets. The suppressed factor λ_u is set to 0.995 to suppress the membrane potential of the firing undesired neurons below the threshold. For the proxy gradient, we adopt the Slayer (Shrestha and Orchard, 2018). Table 1 lists other hyperparameters used.

The following notation is used to describe the network architecture: “FC” stands for a fully-connected layer, “VAD” means Variable Axonal Delay module, “Local” denotes the local skip-connection architecture, and L_Mem implies the use of the suppressed loss in addition to the spike rate loss. For example, Input-128FC-VAD-Local-128FC-VAD-Local-Output + L_{Mem} indicates that there are two dense layers with 128 neurons, each implementing the VAD and Local module. The loss is measured by the spike rate and suppressed membrane potential. Table 2 summarizes the abbreviations for different architectures and methods.

The number of spikes generated from the last layer is compared to the desired spikes in dedicated output nodes, serving as the primary loss measurement. In order to implement the suppressed membrane potential loss function, the model is pre-trained for 20 epochs to generate the target spike trains used for L_Mem definition. For a fair comparison, all the experiments are run for 5 independent trials, and the average performance and standard deviation are reported.

Tests are performed on the speech classification datasets NTIDIDIGITS and Spiking Heidelberg Digits (SHD). Both datasets represent events in the form of spikes, containing rich temporal information that is naturally suited to be directly processed by an SNN. These datasets are considered benchmarks, allowing us to focus on the architecture and learning algorithm of the SNN without considering the spike generation method.

This section demonstrates the benefits of the proposed innovations and assesses the effects of the VAD, Local skip-connection, and Suppressed loss individually to validate their impact on boosting performance. The basic SNN consists of 2 hidden layers, followed by the VAD module, Local skip-connection in each layer, and the suppressed loss module in the readout layer's membrane potential (Figure 2).

1) NTIDIDIGITS. As shown in Table 3, non-spiking approaches such as GRU-RNN and Phased-LSTM (Anumula et al., 2018) achieve 90.90 and 91.25% accuracy, respectively. However, these RNNs rely on the event synthesis algorithm and cannot fully exploit sparse event-based information. Zhang and Li (2019) directly train the spike-train level features with recurrent layers through the ST-RSBP method, and Zhang and Li (2021) further propose the SrSc-SNNs architectures that consist of three self-recurrent layers with skip-connections, training this SNN using backpropagation-based intrinsic plasticity, achieving state-of-the-art (SOTA) performance. We show that with the proposed VAD module, local skip-connection, and suppressed loss, our method achieves 95.30% accuracy with a mean of 95.22% and a standard deviation of 0.08%, making it the best result in this classification task. Furthermore, our model uses the least parameters and is 10× smaller compared to the second-best result.

2) SHD. For this dataset, we compare our methods with recent advancements. In Cramer et al. (2020), the single feed-forward SNN and Recurrent SNN are both trained using BPTT. Their results show that the recurrent architecture outperforms the homogeneous feed-forward architecture in this challenging work, underscoring the potential advantages of intricate SNN designs. Several studies have ventured into specialized SNN architectures. For instance, some explore the effectiveness of the heterogeneous recurrent SNNs (Perez-Nieves et al., 2021), while others delved into attention-based SNNs (Yao et al., 2021). As detailed in Table 3, our proposed method produces a competitive performance of 92.56% in a two-layer fully connected network of 128 neurons each. Notably, this performance is competitive compared to these results that employ the same data processing methods and network architecture. Patiño-Saucedo et al. (2023) introduce axonal delays in tandem with learnable time constants, enabling a reduction in model size to a mere 0.1 M while preserving competitive performance.

Additionally, RadLIF (Bittar and Garner, 2022) combines an adaptive linear LIF neuron with the SG strategy, achieving a performance of 94.62%. This achievement is realized through the utilization of three recurrent spiking layers, each containing 1024 neurons. On the other hand, DCLS, introduced in Hammouamri et al.'s research (Hammouamri et al., 2023), capitalizes on several key innovations. It incorporates learnable position adjustments within the kernel, employs advanced data augmentation techniques (like the 5-channel binning), and incorporates batch normalization methods. As a result, DCLS achieves an accuracy of 95.07% using two feedforward spiking layers, each comprising 256 neurons. Given the sizeable 700-input channel, we mitigated extensive parameter expansion by augmenting the neural network's second layer from 128 to 256 neurons. This strategic adjustment significantly improved performance, yielding a 93.55% accuracy rate.

We delve into the contributions of VAD, Local skip-connection, and Suppressed loss via a comprehensive ablation study (refer to Table 4). Evaluating each method individually on two fully-connected feed-forward SNNs provides the following insights:

Combining VAD and Local skip-connection in the DL128-SNN design yields significant benefits. We clinch state-of-the-art accuracy levels for both datasets. This highlights that the enhanced flexibility provided by VAD truly shines when paired with a richer parameter landscape, as provided by the Local skip-connection. Lastly, supplementing the above with the suppressed loss, Dloss, results in stellar performance: 95.22% for NTIDIDIGITS and 92.56% for SHD.

In this section, we begin by offering a visual representation of the axonal delay distribution (refer to Figure 4) for both datasets. Subsequently, we employ an L2 regularizer on the delay to curtail the magnitude of delay values, effectively reducing the number of delayed time steps.

Utilizing the NTIDIDIGITS dataset as an illustrative example, Figure 4A reveals a delay distribution in the first layer that consistently encompasses both long and short delay neurons. This may imply that certain neurons focus on the initial portion of the input, whereas others concentrate on the latter segment of the input features. To understand the dynamics of the VAD, we inspect the cumulative spike count at the input of the network and compare it to the cumulative spike count at the true decision neuron for four different models, as depicted in Figure 5. For illustrative purposes, we select four different English-speaking digit utterances: “1”, “6”, “7”, and “10”. The figures clearly show that the model without delay gradually increases its prediction as the input spikes come in and starts to do so as soon as input spikes start arriving. Conversely, for the other three models equipped with delay modules, the decision to increase spike count in the true neuron is delayed but then increases more quickly and reaches a higher level. This phenomenon arises from the different neurons introducing varying delays to the spikes, thereby providing the terminal neuron with multi-scale information. This may be interpreted as the VAD-enabled network aggregating all information in the spoken word before triggering a decision using all that information simultaneously. Moreover, we can observe that the models with delay typically have a total of 60 time step latency, which can be measured after the input is over. This is not only related to the delay itself but also to the choice of loss evaluation. As Shrestha et al. (2022) discussed, the spike-based negative log-likelihood loss results in early classification, even 1400 time steps faster than spike-rate based loss evaluation for NTIDIDIGITS datasets. However, the DL-128-SNN-Dloss generates the highest number of spikes for the true neuron compared to the other models, demonstrating its superior ability to learn characterizations.

Subsequently, the L2 loss is employed to confine the range of delay values to provide a more uniform distribution. This leads to a reduction in delay values for some neurons (see Figure 6), aiming to reduce the total latency and investigate whether shorter delays contribute to a better classification system. This is achieved by applying the L2 regularizer to ∑i=1Nd^i. Nevertheless, as demonstrated in Table 4, the inclusion of the additional L2 loss results in a performance decline. This could indicate that the learned distributions achieved through these architectures may already be optimal within the current delay threshold, denoted as θ_d.

The positive impact of local skip-connections on the reset mechanism becomes evident when modulating the refractory scale, symbolized as α_r. We conduct a comparative analysis of performance between two distinct configurations: one labeled as VAD, which encompasses solely the delay model, and the other designated as VAD+Local, which additionally incorporates local skip-connections. As shown in Figure 7, the Local skip-connection maintains high performance across a wider range of refractory scales α_r, while the performance with only the VAD module starts to decline with high values. This observation aligns with our earlier conjecture that larger values of α_r may induce information loss, as the neuron's potential struggles to recover efficiently. In contrast, the presence of local connections mitigates this loss by dynamically triggering spiking events among local neurons. Thus, our Local skip-connection diminishes sensitivity to parameter selection, potentially providing more flexibility to train SNNs for varied tasks, indicating that a consistent alpha value can be effective for different tasks.