Abstract
Recent advances in deep learning have enabled effective interpretation of neural activity patterns from electroencephalogram signals; however, challenges persist in invasive brain signals for cross-day neural decoding and simulation tasks. The inherent non-stationarity of neural dynamics and representational drift across recording sessions fundamentally limit the generalization capabilities of existing approaches. We present AlignNet, a novel framework that establishes cross-modal alignment between spiking patterns and behavioral semantics through U-based representation learning. Our architecture employs hybrid SNN-ANN autoencoders to encode neural spikes and behavior into a shared latent space, where the neural spike autoencoder incorporates multiple neuron nodes following convolution layers, and the behavior autoencoder comprises standard convolution layers. These two representations are optimized through contrastive objectives to achieve session-invariant feature learning. To address cross-day adaptation challenges, we introduce a pretraining strategy leveraging multi-session single monkey experiment data, followed by task-specific fine-tuning for neural decoding and simulation. Comprehensive evaluations demonstrate that AlignNet achieves superior performance under both single-day and cross-day conditions; meanwhile, our pretrained model effectively executes decoding and simulation tasks after fine-tuning. The hybrid SNN-ANN representations exhibit temporal consistency across multi-day recording spikes while retaining behavioral semantics, thereby advancing cross-day neural interface applications.
1 Introduction
In the field of neuroscience, the decoding of neural data has been advanced by deep learning techniques. Animal experiments provide compelling evidence that behavioral states are strongly correlated with specific patterns of brain activity. Research demonstrates that dynamic firing-rate modulations critically influence both the initiation and suppression of motor behaviors (). Deciphering the complex information embedded in neural signals represents a fundamental challenge at the intersection of neuroscience and computational science. To address this challenge, novel computational frameworks have been developed. introduce POYO, a transformer-based architecture that tokenizes neural spike data and utilizes cross-attention to model population-level dynamics across extensive neural recordings. Researchers () propose an unsupervised alignment method designed to leverage shared latent dynamics, facilitating the transfer and reuse of pre-trained generative models across diverse neural datasets.
However, neural decoding encounters challenges in cross-day sessions. Neural non-stationarity () and factors such as postoperative inflammation () induce drift in the behavior-neural signal relationship, compromising decoding stability. To address it, cross-day neural decoding frameworks have been developed. For instance, the Neural Speech Decoding Framework () employs pretrained synthesizers to map neural features to speech waveforms, while domain-invariant representation learning aligns feature distributions across sessions. Neural Encoding Dynamic Sampling (NEDS) () constructs a meta-representation space that captures stable neural dynamics. It employs central kernel alignment (CKA) to quantify cross-day similarity and dynamically calibrates decoder parameters.
Beyond mapping neural data to behavioral outputs, simulating biologically realistic neural dynamics remains a significant yet underexplored challenge in computational neuroscience. To address this gap, designed an artificial neural network that simulates rodent cortical dynamics by integrating multi-scale biological constraints, including spike-timing-dependent plasticity and laminar-specific connectivity pattern. miVAE () proposed a two-stage disentanglement framework that jointly maps neural activity and sensory stimuli into a unified latent space. This approach enables robust identification of cross-modal correlations without requiring subject-specific calibration.
We propose a hybrid SNN-ANN framework that integrates autoencoders and contrastive learning module to complete simultaneous behavioral decoding and neural spike encoding. The proposed architecture leverages the temporal dynamics of spiking neural networks (SNNs) to effectively extract features from neural spike data, while employing artificial neural networks (ANNs) to capture rich representations from corresponding behavioral data. We combine the feature extraction capabilities of U-shaped models for motion trajectories and neural signals with the generalization strengths of contrastive learning. The incorporation of spiking neural networks contributes to adopt biological mechanism when encoding and decoding natural spikes. Evaluation results demonstrate that our framework outperforms existing methods not only in single-day tasks but also in cross-day scenarios.
The main contributions of our work are summarized as follows:
Bidirectional alignment framework: A novel approach for joint embedding of neural spikes and behavior via similarity constraints across trials. It enables discovery of a latent space where stable neural manifold and behavior are bidirectional aligned.
Hybrid SNN-ANN autoencoders: Integration of SNNs (capturing biological fidelity) for robust reconstruction of neural data and ANNs for sequential behavior data utilizes distinct properties of two modalities.
Cross-day generalized capability: A pretraining protocol optimizes on multi-days animal experimental datasets, enhancing generalization of understanding neural signals and corresponding behaviors. The comparison experiments show SOTA ability after finetuning the model on spike-to-behavior decoding and behavior-to-spike simulation tasks.
2 Related works
2.1 Neural signal decoding and simulation
Brain-computer interface (BCI) technology represents a transformative paradigm in human-machine interaction. It enables direct communication between the brain and external devices without relying on conventional neuromuscular pathways (; ). BCI system acquires and decodes neural signals to translate user intentions into actionable commands, generating applied fields, including neurorehabilitation (), assistive technologies (), cognitive augmentation () and immersive communication (). These technique delineates the operational framework and implementation guidelines for BCI protocols, with emphasis on signal acquisition () and neural decoding ().
Signal acquisition constitutes the foundational stage of the BCI system, directly determining the quality and interpretability of subsequent neural data analysis. This process involves the recording of electrophysiological or hemodynamic activity from the brain using either non-invasive modalities, including electroencephalography (EEG) (), functional near-infrared spectroscopy (fNIRS) () and magnetoencephalography (MEG) (), or invasive approaches like electrocorticography (ECoG) () and intracortical microelectrode () arrays. Key technical considerations include electrode placement and density, sampling rate, signal-to-noise ratio optimization, artifact suppression, and the selection of appropriate signal paradigms.
Decoding neural signals into behavioral outputs has developed significant progress in current research. POYO () model introduces transformer framework to enable efficient training across multiple sessions of neural recordings, which advancing the decoding applications. A recent study by () demonstrated that recording just a small part of the total neurons from dorsal premotor cortex can help decode 3D hand movement trajectories with high accuracy. The BraVL () model aligns brain activity, visual images, and textual descriptions in a shared latent space, achieving a high decoding accuracy on untrained new categories. It demonstrates the strong generalization capability of contrastive learning in neural decoding tasks.
Although recent advances in algorithms have demonstrated their potential for neural decoding, cross-day neural decoding remains a challenge due to neural signal shift and behavior state variability (). To solve this problem, researchers worked in different aspects. A new neural speech decoding framework that combines ECoG signal processing with deep learning improves the result of cross-day decoding (). Researchers also designed the NEDS () network to help improve encoding and decoding efficiency at the same time. This network leverages multitask masking, alternating between neural, behavioral, and cross-modal masking during training. This method is able to predict behavior from neural activity and predict neural activity from behavior at the same time. Unlike NEDS, we employ a couple Unet-based architecture combined with contrastive learning to enhance spike-to-behavior accuracy.
Simulating neural activity from a sequence of behavior is a significant study in neuroscience, and breakthroughs in this direction could substantially advance the field of BCIs. Recent work has shown that reinforcement learning can mimic the behavior of freely moving rats and predict neural activity in the sensorimotor striatum and motor cortex (). Researcher developed a neuromorphic tactile system that uses spike timing, especially first-spike timing, to encode dynamic tactile information about touch and grasp (). Furthermore, the NEUSPA modal () successfully integrates Izhikevich neurons with motor unit differentiation to simulate abnormal spike activity in post-stroke spasticity. Moreover, miVAE () employs a two-level disentanglement strategy to map neural activity and visual stimuli into a unified latent space with artifical neural networks(ANN). Compared with miVAE, our hybrid SNN-ANN model incorporates bidirectional alignment to discover neural manifold and build shared latent space; meanwhile, pretraining protocol boost the generalized capability of cross-day spike-to-behavior decoding and behavior-to-spike simulation.
2.2 Deep learning models in BCI
CNNs have been widely applied in neural signal decoding research. For instance, researchers at Purdue University employed a CNN-based framework to implement neural encoding and decoding for dynamic natural vision (). In a parallel development, a hybrid CNN-Transformer architecture was introduced by a research group from the University of Electronic Science and Technology of China (), demonstrating enhanced efficiency in decoding visual neural activity into textual descriptions. Furthermore, the CSM model () successfully decoded macaque finger movement trajectories by incorporating temporal dependencies through CNN structures, effectively correlating current neural states with preceding moments.
Spiking neural networks (SNNs) are biologically-inspired neural networks ; that process information through discrete, asynchronous spikes or action potentials, stimulating the event-driven communication of biological neurons. Compared to ANNs, SNNs work better in event-driven data () and perform efficiently in computer vision tasks . CREST () is an event-based object detection framework. It leverages model with an attention-based bridge, converting spikes to dense features while preserving spatiotemporal dynamics. With lower energy cost, the Spike-Driven Transformer incorporates binary spike communication and spike-driven self-attention (SDSA), achieving high accuracy on the ImageNet-1K dataset ().
Inspired by the biological dynamics of neurons, spiking neural networks have emerged as a pivotal model in computer vision and neuroscience fields . For example, Spike Voxel Coding (SVC) () model utilize SNNs network to solve various 3D computer vision tasks, including classification, detection and segmentation. Recent advances in the Dual-Spike Self-Attention (DSSA)mechanism () and the Self-Backpropagation of Synaptic Modifications (SBP) () enhance performance of the image classification task. The adaptive firing neuron model (AdaFire) also works well in event-driven classification, object detection, and segmentation tasks (). Apart from the work in computer vision tasks, the spiking-based networks have proven to work well to bridge the gap between neuroscience and behavior. The motorSRNN () and STAA-SNN () both prove the high accuracy and efficiency of decoding from electrophysiology to tasks. Moreover, the spiking-based transformers outperform in event-stream tasks () and efficient training process ().
In the field of representation alignment, contrastive learning has emerged as a dominant approach, achieving remarkable performance gains across language (), vision () and multimodal domains (). However, conventional methodologies remain critically dependent on large batches of negative samples, introducing substantial computational overhead and inducing representation bias (). Recent advances mitigate these limitations through asymmetric network architectures and cross-modal alignment techniques (), enhancing the efficacy of unsupervised paradigms. CLIP () and FLAVA () utilize dual-stream encoders to align vision-language embeddings. Subsequent work like CoCa () unifies contrastive and generative objectives, outperforming supervised baselines across downstream tasks.
3 Methods
3.1 Dataset description and preprocessing
In this study, the dataset of Joystick-controlled robotic arm whack-a-mole data from a macaque was obtained from the Brain Science Data Center, Chinese Academy of Sciences1, where the open access data repository are published. All animal procedures were conducted in accordance with protocols approved by the Institutional Animal Care and Use Committee (IACUC) of the Institute of Neuroscience, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences. During the experimental procedure, neural activity was recorded from a macaque performing an eight-direction center-out task, as illustrated in Figure 1. Signals were collected from the macaque's contralateral primary motor cortex (M1). The dataset comprises neural signals acquired at 30 kHz across 64 channels, along with simultaneous 1 kHz two-dimensional joystick voltage signals that serve as the decoding ground truth. Our dataset comprised 32,083 trials collected over 22 days, with neural signals and macaque behavioral signals simultaneously recorded during the experimental procedure. All neural recordings were obtained from a macaque monkey during continuous sessions between April 1, 2025, and April 29, 2025.
Figure 1
The preprocessing pipeline, illustrated in Figure 2, involved the following steps: spike detection, spike matrix reconstruction, behavioral data interpolation, downsampling, and smoothing. Neural preprocessing commenced with spike detection on the raw 64-channel recordings to identify action potentials, converting the continuous data into discrete spike events to improve subsequent feature extraction. Subsequently, a spike matrix was reconstructed by aligning these waveforms, extracting their features, and structuring them into a two-dimensional matrix. The x-axis represents the timestamp, and the y-axis corresponds to the channels, with a sampling rate of 30 kHz for both the spike matrix and behavioral data.
Figure 2
To achieve temporal synchronization, behavioral data were interpolated to match the sampling rate of the neural data, followed by spike matrix reconstruction. The linear interpolation function reshapes the behavioral data to the same sampling rate as the spike data, as described in Equation 1,
In this equation, , α = pj − ⌊pj⌋. m and n denotes the total number of sampling points for raw signal and resampled signal, pj represents the position of j-th sample in the original signal that corresponds to the j-th sample of the target signal. x[⌊pj⌋] and x[⌈pj⌉] denote the lower and higher index of sampling point in the raw signal near the resampled point.
Both neural and behavioral data were then downsampled to a unified sampling rate of 50 Hz. The downsampling equation for neural spikes is given in Equation 2
where Pk denotes spike, k represents the trial number and I indicates the window width. For the behavior signal, the downsampling equation is implemented similarly to the interpolation step, as shown in Equation 1. After aligning the spike and behavior data, a smoothing filter was applied to the behavioral data to enhance the signal-to-noise ratio (SNR). The equation for the Gaussian noise filter is provided in Equation 3
where σ represents the standard deviation, i denotes the i-th sampling points in the signal. The Gaussian smoothing filter reduces noise and high-frequency components while preserving the overall signal shape through weighted averaging with Gaussian weights, thereby better capturing the macaque's behavior.
After the aforementioned preprocessing steps, the final preprocessed data structure comprised a 64-dimensional spike train and a 2-dimensional trajectory per trial. Each trial was segmented into 2-second samples containing 100 temporal points before being input to the network.
3.2 Proposed method
3.2.1 Overall architecture
We propose AlignNet, a novel framework for contrastive learning of alignment of spike-and-behavior patterns. The architecture comprises three core components: a SNN-based autoencoder dedicated to natural spike data, an ANN-based auencoder for behavior data, and a contrastive learning module. Figure 3 illustrates the overview of the AlignNet framework.
Figure 3
The autoencoder performs unsupervised learning by learning an efficient representation (encoding) of input data through training the network to accurately reconstruct its original input. In AlignNet, two autoencoders with unsharable weights are employed to reconstruct natural spikes and behavior. We set U-based network as the basic framework for our dataset, the multi-layer convolutions extract the features of spikes and behavior temporally and spatially; meanwhile, skip connection ensures information reusage for the process of upsampling and deconvolution. Instead of conventional ANNs, we utilize SNNs for encoding and decoding brain activities since such spiking-based networks model neural dynamics more closely, typically composed of multiple layers of spiking neurons. Specifically, in our implementation, we adopt the Leaky Integrate-and-Fire (LIF) spiking neuron nodes (), which will be illustrated in detail later.
The contrastive learning component facilitates the construction of a shared latent space for both natural spikes and behavior, where each data modality is projected into hidden vector representations. This technique operates by contrasting similar (positive) and dissimilar (negative) data samples. Rather than relying on explicit labels, it leverages the inherent structure of the data to learn discriminative features and encourage alignment between the spike and behavior representations. By integrating contrastive learning, the representations of natural spikes and behavior become constrained within a coherent and aligned latent space.
3.2.2 Hybrid SNN-ANN autoencoders
Behavioral experiments involved simultaneous recordings of natural spiking activity and corresponding behavioral sequences during a Center-Out task. In this task, monkeys guided a cursor via a joystick to interact with visual targets. The neural data consisted of spiking signals, while the behavioral data were quantified as the trajectories of the cursor movements across trials. The encoder-decoder framework is grounded in the principle of reconstructing input data from its latent representations. The fundamental operations are defined as:
Specifically, the encoder f(·) transforms the input data x into a lower-dimensional latent representation z, and the decoder g(·) subsequently aims to reconstruct the original input from z.
To facilitate contrastive learning in our framework, the latent vectors derived from both spike and behavior data must share the same dimensionality. We therefore employed a Multi-Layer Perceptron (MLP) to project the behavioral trials into the same dimensional space as the neural dSata prior to the encoding step. This design also aids the U-based encoder in more effectively capturing the temporal dynamics present in both the spiking and behavioral embeddings. As the research objective is to achieve bidirectional mapping–decoding behavior from spikes and simulating spikes based on behavior–we adopted a Mean Squared Error (MSE) loss function to train the U-based Autoencoders.
The encoder architecture sequentially integrates Convolution, Batch Normalization, ReLU activation, and Pooling operations. Spiking data consists of non-negative, multi-channel voltage signals that exhibit temporal variations, capturing diverse neural activities across different brain regions. We utilize a U-based encoder to process trials of natural spike data (), enabling the extraction of both relative temporal patterns within time intervals (S) and spatial correlations across channels (C). The mathematical representation of natural spike encoding is defined as follows:
where is the hidden vector of nature spikes and Bottleneck reshapes the output of k-layer downsample into D-dimension vectors. Specifically, the bottleneck process includes AvgPool denoting 1x1 average pooling with stride 1, W1 representing an 1 × 1 convolution kernel and an ReLU activation function (max(0, x)). Downsample contains a convolution stage and a function of MaxPool denoting 2 × 2 max pooling with stride 2. In the convolution layer, * represents convolution operation and W(i) are 3 × 3 convolution kernels with distinct input and output dimensions. Moreover, BN represents batch normalization and LN denotes the LIF neuron that can be described as:
where τ is the membrane potential time constant, t and n respectively denote the indices of the time step and the n-th layer. Ht − 1, n is the membrane potential from previous time step, It, n denotes the input data at the time step t and the n-th layer, and Vt, n is the updated membrane potential. St, n is the spike sequence through the Heaviside function Θ, in which vth determines whether Vt, n triggers a spike or remains silence. In our research, we fix τ, vth and Vreset as 2, 0.3 and 0 respectively.
Similarly, the process of encoding behavior from a sequence of actions () to hidden vectors () works through k-layer downsample and botteneck. Different with natural spike encoding, LNs are replaced with σ denoting ReLU activation function(max(0, x)).
The decoder is trained to reconstruct input data from latent representations. Both natural spike and behavior decoders consist of 4 steps including Upsampling, Convolution, Batch Normalization and activation. Skip connection is used for creating direct pathways between encoder and decoder layers by concatenating feature maps from the encoding path with corresponding upsampled features in the decoding path. Therefore, the decoder is defined as:
where Upsample denotes k-layer upsampling (transposed convolution, Batch Normalization and activation), Xdownsample is the corresponding encoder-level feature map (skip connection).
3.2.3 Contrastive learning
In AlignNet, contrastive learning paradigm learns representations of behavior and simulating natural spikes by contrasting positive pairs against negative pairs in a latent space. Given an anchor sample, the target is to pull positive samples closer while pushing negative samples farther apart. The negative sample in this study is defined as the non-responding behavior or nature spikes out of one trial. According to the task of behavior decoding and natural spike simulation, multimodal domains are obejective to be aligned. Therefore, we levarage a batch of N spike-behavior pairs , the contrastive learning objective is to reduce the loss of:
where the spike-to-behavior loss is:
and the behavior-to-spike loss is symmetrically defined. f(·) is an embedding function corresponds natural spike encoder or behavior encoder in our framework. τ is a temperature hyperparameter. We constrict the hidden vectors in the latent space by maximizing the similarity of the spike-and-behavior pairs. Consequently, the embedding of natural spikes can be considered as the represatation of behavior, which is decoded into behavior; similarly, the embedding of behavior can be decoded by the natural spike decoder directly to simulate the brain activity.
4 Experiments and results
4.1 Training setting
In the experimental design, the dataset was partitioned into three distinct subsets: pretraining, fine-tuning, and testing. The pretraining phase involves joint optimization of the encoder and decoder to model both neural spiking dynamics and behavioral patterns. Data from 19 consecutive days were allocated for pretraining, during which neural and behavioral representations are learned within a shared latent space to achieve cross-day alignment. This pretrained model is expected to exhibit enhanced generalization capability across different recording days, thereby improving overall model efficiency.
During the fine-tuning stage, we conducted two downstream tasks: neural decoding and natural spike simulation, where all parameters were fine-tuned. Implementation strategies for both tasks are detailed in Figure 4. During the fine-tuning stage, we implement a supervised training paradigm where target labels are guided to predict behavioral trajectories or natural spike patterns. For evaluation, input and output representations are regarded as aligned in the embedding space, since the model's contrastive pretraining establishes consistent latent representations for behavioral and neural dynamics. Consequently, natural spikes are encoded into latent representations via the spike encoder, followed by fine-tuning of the parameters in the behavior decoder to obtain enhanced prediction results.
Figure 4
To evaluate cross-day generalization, we established two experimental protocols: single-day and cross-day. For the single-day task, we split 80% of the data from one day as training set and 20% from on the same day as evaluation set. In the cross-day task, two days of data was used for training and one held-out day for evaluation, assessing generalization capability.
4.2 Performance evaluation
To evaluate performance of our model, we benchmark the model performance against several established models to ensure a comprehensive comparison. In commonly used models, we employ the Wiener Filter, Gated Recurrent Unit (GRU), and vanilla MLP as benchmarks. The Wiener Filter () represents the classical statistical approach, providing an optimal linear estimate under stationary assumptions. For deep learning baselines, we include GRU model to handle the time-series nature of the data and a MLP as a simple yet powerful feed-forward benchmark. The GRU's ability to learn from historical information makes it suitable for modeling the temporal dynamics of neural activity, while the MLP tests the efficacy of treating sequences as fixed-dimensional vectors. This selection allows us to contrast linear, recurrent, and dense non-linear methodologies.
Beyond these comparatively simple benchmarks, we delve into more complex architectures to implement decoding and simulation. Besides UNet structure, we tried to utilize Resnet, which enables the training of extremely deep networks by mitigating the vanishing gradient problem, allowing gradients to flow directly through these identity mappings. Building on this, a hybrid Resnet+UNet model is implemented, where the Resnet encoder captures features at multiple scales, and the UNet decoder pathway facilitates precise, detailed output generation by combining high- and low-level features, which is critical for accurate signal reconstruction.
In our experiments, the evaluation metrics primarily include R-square Error and Pearson correlation coefficient(PCC). R-square Error serves as a key measure for assessing regression performance by quantifying the proportion of variance explained by the model. The equation of R-squared function can be shown as Equation 18.
where yi means the actual value of the i-th observation, ŷi means the predicted value of the i-th observation, ȳ means the mean of actual values.
The PCC evaluates the linear correlation between predicted movements and ground-truth data, with its calculation involving covariance normalization; this metric is also employed in spike simulation tasks to measure the similarity between simulated and natural brain activity patterns, where higher values indicate greater pattern similarity. The equation can be shown as
In this equation, xi and yi means the value of the i-th observation for variable X and Y while and ȳ means the sample mean of variable X and Y.
4.3 Results
The performance of various models on spike-behavior and behavior-spike prediction tasks is summarized in Table 1. We conducted both single-day and cross-day experiments to evaluate the proposed model's decoding capability and generalization performance.
Table 1
| Models | Single-day | Cross-day | ||
|---|---|---|---|---|
| spike → behavior(R2) | behavior → spike(PCC) | spike → behavior(R2) | behavior → spike(PCC) | |
| Wiener Filter () | 0.354 | 0.442 | 0.08 | 0.409 |
| GRU | 0.250 | 0.194 | 0.09 | 0.204 |
| MLP | 0.723 | 0.492 | 0.366 | 0.349 |
| Resnet | 0.413 | 0.323 | 0.327 | 0.201 |
| Resnet+UNet | 0.327 | 0.227 | 0.390 | 0.112 |
| Ours | 0.880 | 0.539 | 0.581 | 0.440 |
The performance of different models under single-day and cross-day conditions.
4.3.1 Single-Day performance
In the single-day experiments, the models were trained and tested on data from the same recording day. Eighty percent of the data were used for training and the remaining twenty percent for testing. This experiment was designed to evaluate the model's ability to learn the relationship between spike activity and behavioral features within one day.
Traditional models, the Wiener filter and GRU, exhibited limited predictive capability. The Wiener filter achieved an R2 of only 0.354 for spike-behavior decoding and a PCC of 0.442 for behavior-spike prediction. The GRU performed even worse, with an R2 of 0.250 and a PCC of 0.194. These results indicate that such models struggle to capture the nonlinear and dynamic relationship between neural signals and behavior. The MLP model performed slightly better, with an R2 value of 0.723 for spike-behavior prediction and a PCC of 0.492 for behavior-spike prediction. However, its performance is still limited by its insufficient ability to capture the complex dynamics of neural signals.
Our proposed model significantly outperformed these baselines. It achieved an R2 of 0.880 for spike-behavior decoding and a PCC of 0.539 for behavior-spike prediction. This improvement can be attributed to the model's integrated U-Net architecture, contrastive learning mechanism, and bidirectional design, which together enable more robust feature extraction and cross-modal alignment. Furthermore, after a single training session, our model can directly perform bidirectional reasoning through fine-tuning. It can adapt to different task requirements without changing the model architecture. This unified framework significantly enhances the model's versatility and deployment efficiency. In contrast, other models usually need to adjust their architectures when dealing with different tasks, which increases the complexity of development and maintenance.
4.3.2 Cross-Day performance
To evaluate model generalization across time, we further conducted cross-day experiments. In this setting, the models were trained on data from two consecutive days and tested on data from a third day. The goal was to test whether models can adapt to new data recorded on different days, where neural activity and behavior may vary.
Traditional models showed a clear degradation in performance. The Wiener filter achieved an R2 of only 0.08 for spike-behavior decoding and a PCC of 0.409 for behavior-spike prediction. The GRU model performed similarly poorly, with an R2 of 0.09 and a PCC of 0.204. These results indicate that the traditional model has significant deficiencies in its cross-day generalization ability. The deep learning models MLP and ResNet + UNet performed slightly better. The cross-day spike-behavior decoding R2 value of MLP was 0.366, and the PCC of behavior-spike prediation was 0.349. The R2 value of ResNet+UNet was 0.390, and the PCC was 0.112. However, their performance in the cross-day task was still limited by the insufficient ability to adapt to the dynamic changes of neural signals.
In contrast, our model demonstrated strong generalization capabilities. It achieved an R2 of 0.581 for spike-behavior decoding and a PCC of 0.440 for behavior-spike prediction. This performance gain is attributed to the model's contrastive pretraining strategy, which aligns neural and behavioral representations across multiple sessions, and its use of hybrid SNN-ANN to extract event-driven and sequential features separately. It is worth noting that our model also maintains the advantage of keeping the same architecture in cross-day tasks. Through one training session and subsequent fine-tuning, bidirectional decoding functionality can be achieved. There is no need to modify the model structure for different tasks. In contrast, traditional methods and other deep learning models often require architectural adjustments and re-training across different tasks, which reduces the efficiency and universality of the system.
4.4 Case analysis
4.4.1 Decoding task
To visualize the model's decoding performance, we plot the decoding outcome graphs in Figure 5. We present the results of three single-day decoding cases and three cross-day decoding cases, with data selected randomly. In each subplot, the upper section shows the variation along the x-axis, while the lower section displays the variation along the y-axis. The red and blue lines represent the ground truth and prediction, respectively. We can clearly observe that in the single-day decoding results, the predicted values align closely with the actual values. To demonstrate the advantages of our model, in addition to the single-day results, we have also randomly selected three cross-day result plots. These include cases with both strong and relatively weaker fitting performance. We can find that our model can roughly depict the movement trajectory of the macaque. Nevertheless, the predicted trajectories confirm the effectiveness of our model in cross-day behavioral decoding tasks.
Figure 5
4.4.2 Simulation task
We present the results of behavioral spike simulation in Figure 6, where the leftmost plot shows the single-day simulation results, while the middle and right plots display cross-day results. The upper and lower subplots represent the ground truth and prediction, respectively, with the x-axis representing timestamps and the y-axis representing channel numbers, in a 100 × 64 dimension. In the single-day results, we observe that the model's predictions closely align with the ground truth across most channels and timestamps, demonstrating the model's high efficiency in single-day tasks. In the two cross-day results, the middle plot shows strong simulation performance in channels 20 to 35 and 1 to 10, but fails to capture spike activity in channels 55 to 64. In the right plot, the simulation achieves better results in channels 1 to 10, while partially reproducing neural activity in channels 20 to 35.
Figure 6
To further compare the firing rate of different channels, we split the 64 channels of each trial into several groups. Every group contains 8 channels, the figure can be shown as Figure 7. In each plot, x axis means the average firing rate of different channels. The equation can be shown as Equation 20.
In this equation, i means the channel index while t means the value of t-th sampling points in i-th channel. The results reveal consistent firing rate patterns across both ground truth and prediction data. Specifically, channels 1–8 and 25–32 exhibit significantly higher firing rates in all experimental conditions, whereas channels 33–64 show substantially lower firing rates in the ground truth data and even more attenuated activity in the predictions. This differential pattern suggests that channels 1–8 and 25–32 may be more strongly correlated with macaque behavioral outputs while 33–64 shows little relationship with macaque behavior cross different days. We suggest that during cross-day training, our model effectively captures the dynamics of high-frequency neural activity but shows limitations in simulating channels with weak neuronal firing. This indicates that while some neurons may lack cross-day consistency or task relevance, our model successfully identifies and learns the simulation patterns of behaviorally-relevant neurons. These findings collectively demonstrate our model's superior capability in cross-day neural decoding tasks.
Figure 7
5 Conclusion
This work introduces an innovative network architecture that synergistically combines the advantages of hybrid SNN-ANN framework and contrastive learning. The proposed model simultaneously handles two key tasks: decoding behavior from neural spikes and simulating spikes from behavioral data. It achieves comparable performance under both single-day and cross-day training settings compared to existing baseline models. SNNs effectively extract features from neural spikes, while ANNs encode information from macaque behavior. The UNet architecture and contrastive learning collaboratively map both types of data into a shared latent space during training. Additional analysis of simulation outputs revealed that while strong inter-day consistency emerged in behaviorally relevant neural channels, others showed no detectable spike activity, suggesting that the model selectively captures task-critical neural representations. These findings highlight the broad utility and robustness of the proposed framework for neural decoding and simulation applications.
Statements
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
The animal study was approved by Center for Excellence in Brain Science and Intelligence Technology, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Neuroscience, Chinese Academy of Sciences, Shanghai, China. The study was conducted in accordance with the local legislation and institutional requirements.
Author contributions
BH: Data curation, Formal analysis, Methodology, Project administration, Resources, Software, Supervision, Validation, Writing – original draft, Writing – review & editing. ZX: Data curation, Resources, Software, Visualization, Writing – original draft. TeZ: Writing – original draft, Data curation, Formal analysis, Visualization. TiZ: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing – review & editing.
Funding
The author(s) declared that financial support was received for this work and/or its publication. This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDB1010302), the National Key Research and Development Program of China (2025ZD0217200), CAS Project for Young Scientists in Basic Research (YSBR-116), Youth Innovation Promotion Association CAS, the Lingang Laboratory Fund (Grant No. LG-GG-202402-06-07, LGL-1987-09), the Shanghai Municipal Science and Technology Project (Grant No. 25ZR1401370, 25LN3200400), and Special Support Project of Guangdong Province (Grant No. 0720240209).
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AldarondoD.MerelJ.MarshallJ. D.HasencleverL.KlibaiteU.GellisA.et al. (2024). A virtual rodent predicts the structure of neural activity across behaviours. Nature632, 594–602. doi: 10.1038/s41586-024-07633-4
2
AwuahW. A.AhluwaliaA.DarkoK.SankerV.TanJ. K.TenkorangP. O.et al. (2024). Bridging minds and machines: the recent advances of brain-computer interfaces in neurological and neurosurgical applications. World Neurosurg. 189, 138–153. doi: 10.1016/j.wneu.2024.05.104
3
AzabouM.AroraV.GaneshV.MaoX.NachimuthuS.MendelsonM.et al. (2023). A unified, scalable framework for neural population decoding. Adv. Neural Inf. Process. Syst. 36, 44937–44956.
4
BashfordL.RosenthalI. A.KellisS.BjånesD.PejsaK.BruntonB. W.et al. (2024). Neural subspaces of imagined movements in parietal cortex remain stable over several years in humans. J. Neural Eng. 21:046059. doi: 10.1088/1741-2552/ad6e19
5
BinnieC.PriorP. (1994). Electroencephalography. J. Neurol. Neurosurg. Psychiatry57, 1308–1319. doi: 10.1136/jnnp.57.11.1308
6
CaoS.-H.HanX.-Y.ZhaoZ.-P.GuJ.-W.JiangT.-Z.YuS. (2025). Hand position fields of neurons in the premotor cortex of macaques during natural reaching. Nat. Commun. 16:3489. doi: 10.1038/s41467-025-58786-3
7
ChenL.KarilanovaS.ChakiS.WenC.WangL.WinbladB.et al. (2024a). Spike timing-based coding in neuromimetic tactile system enables dynamic object classification. Science384, 660–665. doi: 10.1126/science.adf3708
8
ChenX.WangR.Khalilian-GourtaniA.YuL.DuganP.FriedmanD.et al. (2024b). A neural speech decoding framework leveraging deep learning and speech synthesis. Nature Mach. Intellig. 6, 467–480. doi: 10.1038/s42256-024-00824-8
9
DengL.WuY.HuX.LiangL.DingY.LiG.et al. (2020). Rethinking the performance comparison between snns and anns. Neural Netw. 121, 294–307. doi: 10.1016/j.neunet.2019.09.005
10
DongY.WangS.HuangQ.BergR. W.LiG.HeJ. (2023). Neural decoding for intracortical brain-computer interfaces. Cyborg Bionic Syst. 4:0044. doi: 10.34133/cbsystems.0044
11
DuC.FuK.LiJ.HeH. (2023). Decoding visual neural representations by multimodal learning of brain-visual-linguistic features. IEEE Trans. Pattern Anal. Mach. Intell. 45, 10760–10777. doi: 10.1109/TPAMI.2023.3263181
12
FalasconiA.KanodiaH.ArberS. (2025). Dynamic basal ganglia output signals license and suppress forelimb movements. Nature644, 749–758. doi: 10.1038/s41586-025-09066-z
13
FariasS. T.MungasD.ReedB. R.Cahn-WeinerD.JagustW.BaynesK.et al. (2008). The measurement of everyday cognition (ecog): scale development and psychometric properties. Neuropsychology22:531. doi: 10.1037/0894-4105.22.4.531
14
GhasemiS.GračaninD.AzabM. (2024). “Empowering mobility: brain-computer interface for enhancing wheelchair control for individuals with physical disabilities,” in International Conference on Human-Computer Interaction (Cham: Springer), 234–245.
15
GiuntaB.TanJ.LombardoD. (2024). Inflammation and clinical presentation in neurodegenerative disease: a volatile relationship. Neurobiol. Dis. 189:106321.
16
GlaserJ. I.BenjaminA. S.ChowdhuryR. H.PerichM. G.MillerL. E.KordingK. P. (2020). Machine learning for neural decoding. eNeuro 7:ENEURO.0506-19.2020. doi: 10.1523/ENEURO.0506-19.2020
17
GowreesunkerB. V.TewfikA. H.TadipatriV. A.AsheJ.PellizeG.GuptaR. (2011). A subspace approach to learning recurrent features from brain activity. IEEE Trans. Neural Syst. Rehab. Eng. 19, 240–248. doi: 10.1109/TNSRE.2011.2106802
18
JiangP.XiaoC.FuT.BhatiaP.Kass-HoutT.SunJ.et al. (2025). “Bi-level contrastive learning for knowledge-enhanced molecule representations,” in Proceedings of the AAAI Conference on Artificial Intelligence, 352–360.
19
KhorevV.KurkinS.BadarinA.AntipovV.PitsikE.AndreevA.et al. (2024). Review on the use of brain computer interface rehabilitation methods for treating mental and neurological conditions. J. Integr. Neurosci. 23:125. doi: 10.31083/j.jin2307125
20
LiY.LiangF.ZhaoL.CuiY.OuyangW.ShaoJ.et al. (2021). Supervision exists everywhere: a data efficient contrastive language-image pre-training paradigm. arXiv [preprint] arXiv:2110.05208. doi: 10.48550/arXiv.2110.05208
21
LiuT.ChuaY.ZhangY.NingY.WanG.WanZ.et al. (2024). “Proliferation of cosine-tuning in both artificial spiking and cortical neural networks during learning,” in The First Workshop on NeuroAI@ NeurIPS2024.
22
LiuY.YangJ.GuX.ChenY.GuoY.YangG.-Z. (2023). Egofish3d: Egocentric 3d pose estimation from a fisheye camera via self-supervised learning. IEEE Trans. Multimedia25, 8880–8891. doi: 10.1109/TMM.2023.3242551
23
MaassW. (1997). Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10, 1659–1671. doi: 10.1016/S0893-6080(97)00011-7
24
MaoR.ShenA.TangL.ZhouJ. (2025). “Crest: an efficient conjointly-trained spike-driven framework for event-based object detection exploiting spatiotemporal dynamics,” in Proceedings of the AAAI Conference on Artificial Intelligence, 6072–6080.
25
NaseerN.HongK.-S. (2015). fNIRS-based brain-computer interfaces: a review. Front. Hum. Neurosci. 9:3. doi: 10.3389/fnhum.2015.00003
26
PanX.WangM.WuL.LiL. (2021). Contrastive learning for many-to-many multilingual neural machine translation. arXiv [preprint] arXiv:2105.09501. doi: 10.18653/v1/2021.acl-long.21
27
PetrosyanA.SinkinM.LebedevM.OssadtchiA. (2021). Decoding and interpreting cortical signals with a compact convolutional neural network. J. Neural Eng. 18:026019. doi: 10.1088/1741-2552/abe20e
28
ProudfootM.WoolrichM. W.NobreA. C.TurnerM. R. (2014). Magnetoencephalography. Pract. Neurol. 14:336–343. doi: 10.1136/practneurol-2013-000768
29
QiuX.YaoM.ZhangJ.ChouY.QiaoN.ZhouS.et al. (2025). “Efficient 3d recognition with event-driven spike sparse convolution,” in Proceedings of the AAAI Conference on Artificial Intelligence, 20086–20094.
30
ShiX.HaoZ.YuZ. (2024). “SpikingResformer: bridging resnet and vision transformer in spiking neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (Seattle, WA: IEEE), 5610–5619.
31
SinghA.HuR.GoswamiV.CouaironG.GalubaW.RohrbachM.et al. (2022). “FLAVA: A foundational language and vision alignment model,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (New Orleans, LA: IEEE), 15638–15650. doi: 10.1109/CVPR52688.2022.01519
32
SongY.HanL.ZhangT.XuB. (2025). Multiscale fusion enhanced spiking neural network for invasive bci neural signal decoding. Front. Neurosci. 19:1551656. doi: 10.3389/fnins.2025.1551656
33
SunY.ChenX.LiuB.LiangL.WangY.GaoS.et al. (2025). Signal acquisition of brain-computer interfaces: A medical-engineering crossover perspective review. Fundam. Res. 5, 3–16. doi: 10.1016/j.fmre.2024.04.011
34
TangermannM.MüllerK.-R.AertsenA.BirbaumerN.BraunC.BrunnerC.et al. (2012). Review of the BCI competition IV. Front. Neurosci. 6:55. doi: 10.3389/fnins.2012.00055
35
VermaniA.ParkI. M.NassarJ. (2023). “Leveraging generative models for unsupervised alignment of neural time series data,” in The Twelfth International Conference on Learning Representations.
36
WangY.YangX.ZhangX.WangY.PeiW. (2023). Implantable intracortical microelectrodes: reviewing the present with a focus on the future. Microsyst. Nanoeng. 9:7. doi: 10.1038/s41378-022-00451-6
37
WangZ.FangY.CaoJ.RenH.XuR. (2025). “Adaptive calibration: A unified conversion framework of spiking neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, 1583–1591.
38
WeiQ.HanL.ZhangT. (2024). Learning and controlling multiscale dynamics in spiking neural networks using recursive least square modifications. IEEE Trans. Cybern. 54, 4603–4616. doi: 10.1109/TCYB.2023.3343430
39
WenH.ShiJ.ZhangY.LuK.-H.CaoJ.LiuZ. (2018). Neural encoding and decoding with deep learning for dynamic natural vision. Cereb. Cortex28, 4136–4160. doi: 10.1093/cercor/bhx268
40
YanJ.LuoQ.ChenZ.WangZ.GuoX.XieQ.et al. (2025). Spike-based neuromorphic model of spasticity for generation of affected neural activity. IEEE Trans. Neural Syst. Rehab. Eng. 33, 1360–71. doi: 10.1109/TNSRE.2025.3557044
41
YangC.AnZ.CaiL.XuY. (2022). “Mutual contrastive learning for visual representation learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, 3045–3053.
42
YaoM.GaoH.ZhaoG.WangD.LinY.YangZ.et al. (2021). “Temporal-wise attention spiking neural networks for event streams classification,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (Montreal, QC: IEEE), 10221–10230. doi: 10.1109/ICCV48922.2021.01006
43
YaoM.HuJ.ZhouZ.YuanL.TianY.XuB.et al. (2023). Spike-driven transformer. Adv. Neural Inf. Process. Syst. 36, 64043–64058.
44
YaoM.QiuX.HuT.HuJ.ChouY.TianK.et al. (2025). Scaling spike-driven transformer with efficient spike firing approximation training. IEEE Trans. Pattern Analysis Mach. Intellig. 47, 2973–2990. doi: 10.1109/TPAMI.2025.3530246
45
YaoM.RichterO.ZhaoG.QiaoN.XingY.WangD.et al. (2024). Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nat. Commun. 15:4464. doi: 10.1038/s41467-024-47811-6
46
YuJ.WangZ.VasudevanV.YeungL.SeyedhosseiniM.WuY. (2022). COCA: Contrastive captioners are image-text foundation models. arXiv [preprint] arXiv:2205.01917. doi: 10.48550/arXiv.2205.01917
47
ZhangH.JiaoL.YangS.LiH.JiangX.FengJ.et al. (2024). Brain-computer interfaces: the innovative key to unlocking neurological conditions. Int. J. Surg. 110, 5745–5762. doi: 10.1097/JS9.0000000000002022
48
ZhangT.ChengX.JiaS.PooM.- m.ZengY.XuB. (2021a). Self-backpropagation of synaptic modifications elevates the efficiency of spiking and artificial neural networks. Sci. Adv. 7:eabh0146. doi: 10.1126/sciadv.abh0146
49
ZhangT.JiaS.ChengX.XuB. (2021b). Tuning convolutional spiking neural network with biologically plausible reward propagation. IEEE Trans. Neural Netw. Learn. Syst. 33, 7621–7631. doi: 10.1109/TNNLS.2021.3085966
50
ZhangT.LiC.WangG.ZhangM.YuL.XuB. (2023). Research advances and new paradigms for biology-inspired spiking neural networks. J. Electron. Inform. Tech. 5, 2675–2688.
51
ZhangT.YuK.ZhongX.WangH.XuQ.ZhangQ. (2025a). “STAA-SNN: Spatial-temporal attention aggregator for spiking neural networks,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 13959–13969.
52
ZhangY.WangY.AzabouM.AndreA.WangZ.LyuH.et al. (2025b). Neural encoding and decoding at scale. arXiv preprint arXiv:2504.08201.
53
ZhuY.LeiB.SongC.OuyangW.YuS.HuangT. (2025). “Multi-modal latent variables for cross-individual primary visual cortex modeling and analysis,” in Proceedings of the AAAI Conference on Artificial Intelligence, 1228–1236.
54
ZolfaghariM.ZhuY.GehlerP.BroxT. (2021). “CrossCLR: Cross-modal contrastive learning for multi-modal video representations,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 1450–1459.
Summary
Keywords
brain activity simulation, brain-computer interface, contrastive learning, neural decoding, spiking neural network
Citation
Hong B, Xu Z, Zhang T and Zhang T (2026) Bidirectional cross-day alignment of neural spikes and behavior using a hybrid SNN-ANN algorithm. Front. Neurosci. 20:1772958. doi: 10.3389/fnins.2026.1772958
Received
22 December 2025
Revised
05 January 2026
Accepted
15 January 2026
Published
04 February 2026
Volume
20 - 2026
Edited by
Man Yao, Chinese Academy of Sciences (CAS), China
Updates
Copyright
© 2026 Hong, Xu, Zhang and Zhang.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Tielin Zhang, zhangtielin@ion.ac.cn
† These authors have contributed equally to this work
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.