Abstract
Previous work has shown that the dynamical regime of Recurrent Neural Networks (RNNs)—ranging from oscillatory to chaotic and fixed point behavior—can be controlled by the global distribution of weights in connection matrices with statistically independent elements. However, it remains unclear how network dynamics respond to organizational regularities in the weight matrix, as often observed in biological neural networks. Here, we investigate three such regularities: (1) monopolar output weights per neuron, in accordance with Dale’s principle, (2) reciprocal symmetry between neuron pairs, as in Hopfield networks, and (3) modular structure, where strongly connected blocks are embedded in a background of weaker connectivity. These regularities are studied independently, but as functions of the RNN’s general connection strength and its excitatory/inhibitory bias. For this purpose, we construct weight matrices in which the strength of each regularity can be continuously tuned via control parameters, and analyze how key dynamical signatures of the RNN evolve as a function of these parameters. Moreover, using the RNN for actual information processing in a reservoir computing framework, we study how each regularity affects performance. We find that Dale monopolarity and modularity significantly enhance task accuracy, while Hopfield reciprocity tends to reduce it by promoting early saturation, limiting reservoir flexibility.
1 Introduction
Over the past decades, deep learning has achieved remarkable progress (; ), notably through the rise of large language models (). These models are typically based on feedforward architectures, where information flows unidirectionally from input to output layers. In contrast, Recurrent Neural Networks (RNNs) include feedback connections, enabling them to function as autonomous dynamical systems () that sustain neural activity even without ongoing external input.
RNNs exhibit certain “universal” properties—such as the ability to approximate arbitrary functions () or general dynamical systems ()—which, alongside other strengths, have spurred interest in their fine-grained behavior. For example, they can preserve information from temporally extended input sequences (; ; ; ; ; ) and learn effective internal representations by balancing compression and expansion of information ().
A further key research theme concerns the control of RNN dynamics, including how internal and external noise shape network behavior (; ; ; ; ; ; ; ). RNNs have also been proposed as models for neural computation in the brain (). Notably, sparse RNNs—with a low average node degree, resembling biological circuits ()—have shown improved capacity for information storage (; ; ; ).
In our earlier work, we systematically explored the interplay between network structure and dynamics, beginning with three-neuron motifs (). We later showed how the weight distribution’s width , the connection density , and the balance between excitation and inhibition can be tuned to control the dynamics of large, autonomously active RNNs (; ). We also investigated noise-induced resonance phenomena in such systems (; ; ; ; ; ; ).
Most studies—including our own—have assumed statistically independent weight matrix elements, drawn from fixed distributions and assigned randomly. While analytically convenient, this assumption does not reflect the structural regularities seen in biological networks. Real neural systems exhibit highly non-random connectivity, shaped by development, functional demands, and evolutionary constraints.
The potential benefits of incorporating structural regularities into neural architectures have increasingly been recognized in the literature. Several recent developments have explored this direction, aiming to move beyond fully random or uniform connectivity. For example, Capsule Networks introduced by Hinton and colleagues (; ) implement local groups of neurons—capsules—that preserve part-whole relationships via structured routing mechanisms. In transformer models, architectural variants have been proposed that introduce explicit modularity or routing constraints to enhance interpretability and scalability (; ; ). Moreover, a number of studies have examined recurrent networks with biologically inspired topology, exploring the impact of modularity, reciprocity, or Dale-like constraints on network dynamics and learning performance (; ; ).
Recent theoretical work has further deepened our understanding of how structured connectivity influences recurrent network dynamics. For instance, studies have examined how the number of effective degrees of freedom and the resulting low-dimensional organization of neural trajectories depend on architectural constraints and coupling statistics (; ). Other analyses have characterized dynamical regimes in structured or partially symmetric networks, including glassy attractor states and transitions between ordered and chaotic phases (; ). Together, these works highlight that architectural regularities do not merely stabilize dynamics, but can fundamentally shape the computational landscape of recurrent systems.
These efforts highlight a growing consensus that structural features—long regarded as biological idiosyncrasies—may in fact play a functional role in shaping the computational behavior of artificial neural systems. Against this background, we systematically examine the isolated effects of three such regularities:
First, biological neurons follow Dale’s Principle, meaning each neuron is either excitatory or inhibitory, but not both (; ).
Second, neural circuits exhibit an increased likelihood of reciprocal connections: if neuron A projects to B, B is more likely to project back to A (; ). This bidirectional coupling introduces local symmetries that may stabilize attractor states and support mutual reinforcement.
Third, the brain is modular, comprising groups of neurons more densely connected within than between groups (; ). Such organization appears across scales, from cortical microcircuits to large-scale areas, and enables specialized yet integrative processing.
In the following, we study how each of these biologically inspired regularities affects the dynamical regime of RNNs, using a minimal implementation in which their respective strength can be continuously tuned (compare Figure 1).
FIGURE 1
The first regularity, called Dale Homogeneity, is controlled via a continuous parameter , indicating the degree to which a neuron’s outputs are consistently excitatory or inhibitory. At , polarities are mixed; at , all neurons are strictly monopolar.
The second regularity, Hopfield Reciprocity, is controlled by a parameter . For , weights and are independent; for , they are identical, as in Hopfield networks.
The third regularity is Modularity, parameterized by . A network with lacks modular structure, whereas introduces strongly connected quadratic blocks with standard deviation , embedded in a weaker background with . As the degree of modularity increases, and are adjusted so that the global width remains constant.
While , , and control the strength of structural regularities, we assess their impact on network dynamics using deterministic neurons with activation and three dynamical measures: , , and .
The fluctuation is the standard deviation of neuron outputs, averaged over neurons and time. It quantifies spontaneous activity; excessively large would hinder reliable computation.
The covariance is defined as the product , averaged over neuron pairs and time. In a globally oscillatory state, , in a chaotic state , and in a fixed point state .
Finally, the nonlinearity is the average operating regime of neurons. At , neurons operate linearly; at , they are saturated and behave digitally.
While the primary focus of this study lies on how organizational regularities shape the intrinsic dynamics of RNNs, it is natural to ask whether these structural features also affect the network’s information processing performance. To address this, we embed the RNN into a reservoir computing framework and examine how the accuracy in simple benchmark tasks varies as a function of the regularity control parameters.
The following sections describe the construction of the weight matrices, the simulation setup for measuring dynamical indicators, and the test tasks used to evaluate computational performance.
2 Methods
2.1 General simulation setup
The overall workflow of our investigation consists of the following steps:
First, we set the control parameters, including the distribution parameters—width , density , and excitatory-inhibitory balance —as well as the regularity parameters: Dale homogeneity , Hopfield reciprocity , and modularity .
Second, we generate a set of random square weight matrices that satisfy these specifications. Their properties are verified by empirically evaluating their statistical features.
Third, for each matrix, we simulate the spontaneous dynamics of the corresponding RNN. The network is initialized in a random state and then run freely for a large number of time steps. We refer to the resulting time series of neural activations as the output stream.
Fourth, we compute the dynamical measures—fluctuation , covariance , and nonlinearity —for each output stream, resulting in samples per measure. For visualization, we average these values across the ensemble of networks.
In addition to analyzing intrinsic dynamics, we also examine how the regularity parameters affect information processing. For this, the RNN is embedded into a reservoir computing framework: it receives input via a fixed input matrix, and its output stream is passed to a trainable readout layer. However, in the first part of the study, we set the input matrix to zero, so the RNN runs autonomously without external input, and the readout is not used. Nonetheless, we describe the full reservoir architecture in the following for completeness.
2.2 Design of reservoir computer (RC)
The RC consists of an input layer, a recurrent reservoir, and a readout layer. The input data comprises consecutive episodes, each corresponding, for example, to a pattern to be classified. This data stream is fed into the reservoir and circulates through the system while propagating toward the output.
At each time step , the input layer receives parallel signals . These are linearly transformed by the input matrix of size and injected into the reservoir, as described by Equation 1. Each input episode spans time steps.
The input layer consists solely of the matrix and is therefore purely linear. Its elements are drawn independently from a normal distribution with zero mean and standard deviation . To study autonomous RNN dynamics, we set , effectively decoupling the reservoir from external input.
The reservoir comprises recurrently connected neurons with activation. At each time step, all neuron states are updated in parallel. Each neuron receives a bias term , input from the external signals via , and recurrent input via the weight matrix (see Equation 1). Initial states are drawn uniformly from and kept fixed for repeated simulations of the same reservoir. Different weight matrices receive independent initial states.
The readout layer performs an affine-linear transformation of the reservoir states using a output matrix and a bias vector , as in Equation 2. These parameters are trained via the pseudoinverse method (see below).
In the sequence generation task, which serves as the main information processing benchmark in this work, the continuous outputs of the readout layer directly form the vectors of the output sequence. In classification tasks, by contrast, the represent soft votes that are converted into discrete predicted class labels using the argmax function (see Equation 3). This final step introduces a nonlinearity that sharpens the class boundaries in the output space.
In summary, the RC is governed by the following equations.
2.3 Sequence generation task
In this task, the reservoir computer functions as a deterministic system that maps an input sequence of real-valued vectors onto a corresponding output sequence :Here, and denote the temporal lengths of the input and output sequences, and and their respective vector dimensions.
At the beginning of each episode, a randomly chosen input sequence from one of discrete classes is fed into the network, driving it into a class-specific internal configuration or ‘priming state’. After the input ends, the reservoir evolves autonomously through a sequence of internal states , which the linear readout layer is trained to map onto the corresponding target sequence .
Since the system is strictly deterministic, it will always produce the same trajectory for each distinct priming state. Provided that the induced trajectory is not a cyclic attractor with a period shorter than , the readout should in principle be able to convert the trajectory into the desired output sequence.
However, because the reservoir is not reset at the beginning of each episode, residues of previous states may persist, such that the priming state is not exactly identical every time a given class is selected. If the reservoir operates in a chaotic regime, these small differences can be amplified, producing an effectively unpredictable trajectory that cannot be mapped to the correct target.
In practice, the target mapping also fails if the reservoir neurons enter the saturated ‘digital’ state, since the resulting trajectory is insufficiently rich and lacks the necessary high-dimensional diversity.
Successful performance therefore requires a balance between stability and richness: the reservoir must forget prior excitation rapidly enough to respond reproducibly to identical inputs, while still maintaining sufficient temporal and spatial diversity across neurons.
To systematically explore the influence of network parameters under controlled conditions, we employ a minimal task configuration with , , , and . The class-specific input sequences and their target sequences are predefined, with all vector components drawn independently from a uniform distribution over . In each episode, one of the cases is selected at random.
2.4 Optimal readout layer using pseudoinverse
The optimal weights and biases of the readout layer can be efficiently computed with the method of the pseudoinverse, based on the sequence of reservoir states and the target output (Compare, for example, Section 3.4. in ()). Following these ideas, we proceed as follows:
Let be the matrix of reservoir states directly after each input episode, where is the total number of episodes and is the number of reservoir neurons. Let be the matrix of target output states, where is the number of output units.
To account for biases in the readout layer, a column of ones is appended to , resulting in the matrix :where is a column vector of ones.
The weights and biases of the readout layer are computed by solving the following equation using the pseudoinverse of :where is the Moore-Penrose pseudoinverse of , and contains both the readout weights and the biases.
To compute the pseudoinverse, we first perform a singular value decomposition (SVD) of :where is a unitary matrix, is a diagonal matrix containing the singular values, and is the transpose of a unitary matrix.
The pseudoinverse of is computed as:where is the pseudoinverse of the diagonal matrix . The pseudoinverse is obtained by taking the reciprocal of all non-zero singular values in and leaving zeros unchanged.
Finally, after inserting into Equation 4, the optimal readout weights and biases are extracted from the extended matrix aswhere the first rows of define the readout weights and the last row defines the biases.
2.5 Generation of weight matrices with homogeneity, reciprocity, or modularity
The generation of weight matrices is based on the six control parameters defined in the Introduction. Depending on the selected values of homogeneity , reciprocity , and modularity , different structural features are incorporated into otherwise random matrices.
We begin by generating a matrix of magnitudes , where each entry is drawn from a normal distribution with zero mean and standard deviation , then made positive via .
A binary mask matrix is created, where each element is set to 1 with probability . This defines the sparsity structure.
A sign matrix is generated, with assigned with probability to control the excitatory-inhibitory balance.
The elementwise product of magnitude, mask, and sign matrices yields the pure weight matrix , which serves as the default base.
To implement homogeneity (, ), we generate a Dale-conform matrix in which each column (corresponding to one sending neuron) has a uniform sign. The final weight matrix is then obtained by stochastic interpolation: with probability , and with probability .
The Dale matrix uses the same magnitudes and mask as , but assigns each column a fixed sign (either all or all ), again with controlling the excitatory fraction.
To implement reciprocity (, ), we generate a symmetric Hopfield-like matrix by copying the upper triangle and diagonal of into the lower triangle. The final matrix is again obtained via stochastic interpolation: with probability , and otherwise.
To implement modularity , we divide the matrix into regular square blocks of size , grouping the neurons into modules. Each block is randomly assigned to the “strong” or “weak” class, with the probability of “strong” given by . The realized fractions and are determined empirically after assignment.
Weak blocks are filled with Gaussian values of standard deviation , where . Strong blocks use standard deviation , with
to ensure that the total matrix standard deviation remains .
For , both scaling factors are equal to one; for , the weak blocks vanish and .
Except for the extreme case , where the entries in all weak blocks vanish, the resulting matrix is fully dense . To impose the desired balance , all elements are made positive and then flipped to negative with probability .
For clarity, we never apply homogeneity , reciprocity and modularity simultaneously.
2.6 Evaluation of weight matrices
In addition to the control parameters , we define corresponding empirical parameters that quantify the actual density, balance, homogeneity, and reciprocity of a given weight matrix .
The empirical density is computed as the fraction of non-zero elements:
The empirical balance measures the ratio of excitatory to inhibitory weights:
The empirical homogeneity is computed column-wise. For each column , we calculatewhere and refer to the number of positive and negative entries in column . If all elements in the column are zero, we set . The global homogeneity is the mean over all columns:
The empirical reciprocity is computed from all off-diagonal pairs in the upper triangle:If both and are zero, we define . The matrix-level reciprocity is the average over all such pairs:
For each set of control parameters , we generate an ensemble of weight matrices and compute the empirical parameters as averages over the corresponding values from each individual matrix.
2.7 Fluctuation measure
The neural fluctuation measure quantifies the average temporal variability of reservoir activations. For each neuron , we compute the standard deviation of its activation time series . The global fluctuation is defined as the mean over the standard deviations of the individual neurons:
Since -neurons produce outputs in , the fluctuation lies in . A value of indicates a resting or fixed point state, while corresponds to perfect two-state oscillation (e.g., alternating between and ).
2.8 Covariance measure
To assess temporal covariances, we compute the average product of the activation of neuron at time and neuron at time :
Unlike the Pearson correlation coefficient, we deliberately avoid subtracting the mean or normalizing by the standard deviations. This ensures that the matrix elements remain well-defined even when one or both signals are constant, as in a fixed point state.
The global covariance measure is defined as the average over all neuron pairs, without differentiating between diagonal and off-diagonal elements:
Owing to the bounded output of the neurons, the covariance values always lie within the range .
2.9 Nonlinearity measure
The shape of the activation distribution reflects whether the reservoir operates in a linear or nonlinear regime. A central peak at indicates a linear regime; two peaks near indicate saturation and thus nonlinearity.
We define a nonlinearity measure
based on the fractions of neural activations falling into the following intervals:
The resulting measure distinguishes three regimes: for linear operation, for intermediate or flat activation, and for saturated, digital-like behavior.
This intuitive yet robust definition proved most effective among several tested alternatives. It captures the essential qualitative transition in from unimodal (linear) to bimodal (nonlinear) distributions, as highlighted in earlier studies.
2.10 Accuracy measure
In the sequence generation task, we evaluate performance by comparing the actual output sequences of the readout layer with the corresponding target sequences , and compute the root-mean-square error . This error is normalized by the standard deviation of the target sequences.
To obtain an accuracy measure , we define:
Note that when the RMS error is comparable to the variability of the target data, and when the output matches the target exactly.
In classification tasks, the accuracy is simply defined as the fraction of correctly predicted class labels.
3 Results
3.1 Validation of control parameters
As described in the Methods section, we generate weight matrices with prescribed values for connection density , excitatory/inhibitory balance , Hopfield reciprocity , Dale homogeneity , and modularity . To verify that the generation procedures operate as intended, we compute the corresponding empirical parameters directly from the generated matrices. Throughout this paper, lowercase letters denote the prescribed control parameters, while uppercase letters refer to their empirically measured counterparts. Validation of the modular structure—additionally dependent on block size and the fraction of ‘strong’ blocks—is addressed separately (see below).
For validation, we first fix all control parameters to standard values. Then, one parameter is varied across its entire admissible range while the others remain fixed. During this one-dimensional scan, we compute all empirical quantities as functions of (Figures 2a–e). Ideally, each control parameter should primarily affect its corresponding empirical statistic , without significantly altering the others. However, certain interdependencies are inevitable due to shared structural properties.
FIGURE 2
As shown in panel (a), the prescribed density directly controls the empirical density , with negligible effect on the balance . However, it also influences reciprocity and homogeneity to a minor extent.
Similarly, the prescribed balance determines the empirical balance , while leaving unaffected (panel b). Nonetheless, and increase as the system becomes more unbalanced. Even for perfectly balanced matrices at , there exists a minimal, unavoidable level of reciprocity and homogeneity due to the Gaussian weight distribution.
Varying the prescribed Hopfield reciprocity results in a nearly linear increase of the empirical reciprocity , while the other measures remain unaffected (panel c).
Increasing the prescribed Dale homogeneity leads to a monotonic rise in empirical homogeneity , along with a slight increase in balance . Reciprocity and density remain essentially unchanged (panel d).
Finally, the prescribed modularity has virtually no effect on any of the empirical measures , except at the extreme value (panel e).
To validate the modularity construction, we consider a weight matrix with parameters and . The block size is set to , and the fraction of strong blocks to . We then gradually increase the modularity parameter from 0.2 to 0.8, and compute the histogram of matrix elements for each value (panel f).
A semi-logarithmic plot reveals that the resulting distributions are mixtures of two Gaussians with distinct standard deviations. As expected, the mixture preserves the global distribution width (STD), which is explicitly shown in the legend of panel (f).
Note that the weight distribution of the connectivity matrix becomes a true Gaussian mixture only for intermediate modularity parameters . In the one extreme case , the distributions of the weak and strong blocks have identical Gaussian widths, and thus no block structure exists. As increases, the distribution width of the weak blocks gradually decreases relative to that of the strong blocks. In the opposite extreme case , the weak blocks have zero distribution width, which effectively means that these connections are no longer present.
3.2 RNN phase diagrams
In this section, we examine an RNN consisting of 50 neurons, each randomly connected to all others with a full connection density of . We analyze how the network’s dynamical and information-processing properties vary as a function of the excitatory–inhibitory balance and the width of the Gaussian distribution of connection strengths. The results are presented as “phase diagrams”, where selected quantities are color-coded in the – plane (see Figure 4).
The fluctuation measure quantifies the mean amplitude of the temporal variations in neural activation.
The nonlinearity indicates whether neurons operate predominantly in the linear regime of the sigmoidal transfer function or in the saturated nonlinear regime .
The covariance measure is not a Pearson correlation coefficient, since we deliberately avoid subtracting the means and do not normalize by the variances of the signals. For our specific network of neurons, this definition of leads to a smooth transition from in a globally oscillatory regime, through in a quiescent or irregularly fluctuating regime, to in a global fixed point regime (compare Figure 8f). In combination with the fluctuation measure , this allows us to identify the dominant dynamical regime of the reservoir, as shown previously (Figure 1 in ()).
The accuracy quantifies the performance of the RNN when used as the recurrent core of a reservoir computer (RC). Although the result naturally depends on the specific task, we restrict our analysis here to a single example (Figure 3), in which the RC is driven by different input sequences and required to generate predefined target output sequences. This task is particularly sensitive to the dynamical regime of the RNN, since the state trajectory must remain input-controlled and sufficiently regular over multiple updates, without entering the saturation regime or being dominated by spontaneous irregular fluctuations.
FIGURE 3
3.2.1 Free-running RNN
In the free-running RNN (upper row in Figure 4), we identify four characteristic dynamical regimes, most clearly visible in the nonlinearity phase diagram :
FIGURE 4
FIGURE 5
In the lower central part of the phase plane lies the quiescent region , roughly shaped like ‘Mt Fuji’. In this region, at , the neurons are virtually unconnected. Therefore all activations remain at very small values, determined only by the individual biases. As indicated by the nonlinearity measure , the neurons operate here in the linear regime, near the center of the tanh activation function. The temporal constancy of the activations is reflected in the fluctuation measure . While the Pearson correlation coefficient would diverge for constant zero signals, our non-normalized covariance measure simply yields in this case.
In the right wing of the phase plane lies the fixed point regime . Also marked by temporally constant activations, it exhibits fluctuations . However, due to strong coupling strengths and predominantly excitatory weights , a positive feedback loop drives the network into high-activation global fixed points, where each neuron becomes trapped in either the positive or negative saturation of the activation function. As a result, the nonlinearity measure indicates that the neurons now operate in a digital regime, and the covariance measure yields , showing that the neurons retain the same digital value over time.
In the left wing of the phase plane lies the oscillatory regime . Here, strong , predominantly inhibitory coupling leads to global periodic flips of all neurons between the two saturation states. Consequently, the covariance reflects activation values of opposite sign between successive time steps. The fluctuation indicates high-amplitude temporal variation, and the nonlinearity shows that the neurons operate digitally.
In the upper central part of the phase plane lies the chaotic regime . The dynamics here are also driven by strong mutual couplings , but now the approximate balance between excitatory and inhibitory connections leads to much more complex and irregular behavior—both over time and across neurons. This is reflected in vanishing covariances . The neurons operate in a mixed linear, intermediate, or digital regime, such that the nonlinearity measure lies somewhere between and . The corresponding temporal fluctuations , on average, are smaller than in the oscillatory regime.
3.2.2 RNN with input signals
Next, while the RNN is continuously updating, we feed in two time-dependent input signals related to a computational task. For this purpose, we use a dense input matrix , so that every neuron receives both inputs. The elements of are drawn from a Gaussian distribution with zero mean and a standard deviation of 0.3. The two input signals range between and , and all further details are described in the Methods section.
We find that the injection of inputs has only a very weak effect on the phase diagrams of , , and (see second row in Figure 4). Only the fluctuation level in the quiescent regime is slightly elevated compared to the input-free case, which is to be expected.
We also add a readout layer, optimized by the method of the pseudo-inverse, which transforms the global time-dependent states of the RNN into output signals. In our case, the readout layer is optimized for a sequence generation task (see Methods section for details), and the performance of the resulting reservoir computer is measured by an accuracy value that ranges between zero and one (right-most phase diagram in row 2).
We find that the accuracy remains close to one throughout the entire quiescent regime . As shown in a previous publication, this regime is well-suited for many types of tasks, as the RNN states are then primarily determined by the input, not by spontaneous internal dynamics.
The accuracy drops considerably within the chaotic regime , where the irregular and unpredictable dynamics of the RNN interfere with the execution of the computational task.
A strong reduction in accuracy is also observed in the upper and outer parts of the oscillatory and fixed point regimes. There, the autonomous RNN dynamics are predictable, but the neurons are driven so deeply into the saturation regime that task-related computations can no longer be carried out.
Remarkably, the accuracy remains close to one in the narrow regions between the chaotic regime and the neighboring oscillatory or fixed point regimes. These two ‘edges of chaos’ thus prove suitable for task-related computation; however, they become increasingly narrow as the recurrent coupling strength is increased.
3.2.3 Effect of strong hopfield reciprocity
Next, we leave the reservoir computer unchanged, with the only modification being the introduction of a relatively strong degree of Hopfield reciprocity, , into the RNN’s weight matrix (see third row in Figure 4).
Comparing the nonlinearity phase diagram at with the corresponding diagram at , we observe a significant shrinking of the quiescent regime and an increase in nonlinearity within the chaotic regime .
No strong differences are observed between the fluctuation phase diagrams and or between the covariance phase diagrams and .
Finally, the accuracy phase diagram shows that performance in the chaotic regime has dropped to levels even below those of the system. This suggests that Hopfield reciprocity can, at least in certain computational tasks, have a detrimental effect on the performance of a reservoir computer.
3.2.4 Effect of strong dale homogeneity
Starting again from the standard case without any structural regularities , we now set the Dale homogeneity parameter to and recompute the four phase diagrams (see fourth row in Figure 4).
The diagrams for nonlinearity, fluctuation, and covariance at resemble those observed at more closely than the standard case.
However, the accuracy diagram shows that Dale homogeneity improves task performance even beyond the standard case . In particular, within the chaotic regime , the reservoir computer can now tolerate significantly higher coupling strengths .
3.2.5 Effect of strong modularity
Finally, we set the modularity parameter to , using block sizes of neurons and a strong block fraction of (see Methods for details). Modularity exerts a pronounced influence on all phase diagrams (see fifth row in Figure 4):
The nonlinearity and covariance diagrams indicate that the oscillatory and fixed point regimes now occupy only a narrow region of phase space, restricted to highly unbalanced weights and strong coupling .
Meanwhile, the previously chaotic regime is replaced by a broad region in which both nonlinearity and fluctuations remain moderate, while low covariance values still suggest irregular (non-repetitive) dynamics.
Most strikingly, the accuracy now reaches very high levels across almost the entire phase diagram. This indicates that modularity renders even formerly unproductive regimes—oscillatory, fixed point, and chaotic—computationally useful.
3.3 Effect of gradually increasing regularity
We now examine how a gradual increase of the organizational regularity parameters affects the dynamical quantities and the computational performance (Figure 6). All simulations were performed using an RNN with neurons. We selected three representative combinations of the fundamental control parameters, width and balance , corresponding to three specific points in the – plane (indicated by crosses in the third panel of the top row in Figure 4):
FIGURE 6
Phase point A (at ) represents a balanced network located near the transition between quiescent and chaotic dynamics.
Phase point B (at ) shares the same moderate connection width as point A but is positioned close to the border of the fixed point regime.
Phase point C (at ) is again balanced but lies deeper within the chaotic regime.
The RNN is used throughout as a reservoir in the sequence generation task. While one regularity parameter is varied over its full range [0,1], the others are held at zero.
When the Hopfield reciprocity is increased (panels a,b,c), the most prominent effects—consistent across all three phase points—are a pronounced rise in nonlinearity (green) and a small but significant decline in accuracy (blue). It is also noteworthy that in phase point C, the fluctuation (orange) decreases while the nonlinearity (green) increases, demonstrating that these two measures are not trivially related. The covariance (red) shows hardly any change with reciprocity .
Increasing Dale homogeneity (panels d,e,f) leads to a clear gain in accuracy (blue) across all phase points, indicating that this organizational regularity supports certain types of information processing. The other dynamical quantities show minor, but not entirely consistent, changes as a function of .
Increasing modularity with a block size (panels g,h,i) produces particularly strong effects: the nonlinearity (green) is markedly reduced across all three phase points. At the same time, the fluctuation (orange) decreases in phase points A and B. Most importantly, the accuracy is significantly enhanced by modularity in all cases.
When the block size is reduced to (panels j,k,l), the weight matrix no longer contains distinct blocks. In this case, increasing merely changes the weight distribution from a single Gaussian to a mixture with unchanged total standard deviation. In phase point A, this still yields a modest increase of accuracy (blue), though the dynamical changes are less pronounced compared to the block-structured case. For the other two phase points, no further improvement of accuracy is observed.
3.4 Effect of regularities on neuron activations and pearson correlations
To examine how the regularity parameters influence reservoir dynamics, we use the system in phase point C (without input) and simulate the time series of neural activations (Figure 7, first row). We then analyze the correlations between activations across the reservoir. In contrast to the covariance measure , which was defined as the unnormalized average product of activations at subsequent time steps , we now compute the matrix of instantaneous Pearson correlation coefficients. This means that the mean activity is subtracted and the resulting products are normalized by the variance. We perform the same computation for systems with weak additive noise (third rows). The analysis is carried out for the standard network as well as for each of the three organizational regularities.
FIGURE 7
Without regularities, the activations are highly irregular across both neurons and time (a). As a result, the instantaneous correlations are weak (b).
With maximal Hopfield reciprocity, the network settles, after a short transient, into an attractor state where many neurons oscillate with period two and large amplitudes (some out of phase with others), while others remain in a fixed point state (d). All corresponding Pearson matrix elements exhibit values close to or , including some spurious entries when neurons have near-zero variance (e).
With maximal Dale homogeneity, the neural activations display quasi-periodic collective fluctuations. Most neurons tend to share similar activation signs within each temporal band, while phase shifts and irregularities prevent perfect periodicity (g). Consequently, the Pearson matrix contains many moderately positive entries (h).
Finally, with strong modularity, the system behaves in a more heterogeneous manner. Groups of neurons exhibit longer-period, synchronous oscillations, whereas others show irregular activity (j). The Pearson matrix (k) now spans the full range of possible values between and .
We now repeat the correlation analysis while adding noise to the reservoir. As we have demonstrated in earlier work (), this prevents the reservoir from becoming permanently trapped in a single attractor throughout the simulation. For this purpose, statistically independent Gaussian random values with zero mean and a standard deviation of 0.1 are added to the total input of each individual neuron in each time step.
The added noise has hardly any effect on the system with Dale homogeneity (i) and causes only a slight reduction in the amplitudes of the correlation coefficients in the system without regularities (c) and in the modular system (L). In contrast, for the Hopfield-like system, the variability introduced by the noise eliminates the spurious perfect correlations previously seen in (f). It now becomes apparent that this regularity induces strong correlations—of either sign—between specific pairs and blocks of neurons.
3.5 Supplemental analyses
3.5.1 Patches classification task
We also tested how modularity affects performance in a completely different type of task. For this purpose, 36 patches from two distinct classes were randomly distributed within a two-dimensional input plane (Figure 8a). The coordinates of a random point in this plane were fed into the reservoir at the beginning of each episode. The goal of the reservoir computer was to predict the class label of the continuous input point, and the accuracy was computed as the fraction of correctly classified labels.
FIGURE 8
Although the performance gain is not as strong as in the sequence generation task, the accuracy increases monotonically from near chance level to a significantly higher value when the modularity parameter (at block size ) is tuned from zero to one (Figure 8b), focusing on phase point C.
3.5.2 Larger reservoir
Returning to the standard sequence generation task, we verified the performance gain due to modularity also in a larger reservoir, again focusing on phase point C. As the number of neurons was doubled from 50 to 100, we made both cases more comparable by reducing at the same time the connection width by a factor of .
Recomputing the dynamical quantities and the accuracy as functions of the modularity parameter , we found the same trends as in the smaller reservoir: the nonlinearity (green) and the fluctuation (orange) are again strongly reduced, while the accuracy is significantly enhanced by modularity (Figure 8c).
3.5.3 Chaotic regimes
To demonstrate that within the ‘chaotic regime’ (CR) of phase point C, the temporal evolution of the reservoir exhibits sensitive dependence on initial conditions, we compare the neural activations for two slightly different starting states (Figure 8d). In the second run (middle panel), the initial activation of neuron 0 is increased by , while all other activations remain identical to those in the first run (left panel). The first noticeable differences (right panel) appear after approximately 45 time steps, after which the two state trajectories diverge rapidly.
3.5.4 Sequential updates
The original motivation of Hopfield networks was to create an attractor landscape of fixed points, each representing a stored pattern. In contrast, our networks with Hopfield-like reciprocal connections predominantly exhibit oscillatory behavior. This difference arises from our use of simultaneous updates of all neuron states at each time step, whereas the original Hopfield model employs sequential updates. To demonstrate this effect, we repeated the simulation shown in Figure 7d, where the RNN was at phase point C with a Hopfield reciprocity parameter of , yet this time with sequential updates. As expected, changing the update scheme results in fixed point rather than oscillatory attractors (Figure 8e)
3.5.5 Diagonal and off-diagonal covariances
In our definition of the covariance measure , we do not distinguish between diagonal and off-diagonal contributions. To illustrate that both behave similarly, we simulated a reservoir of 50 neurons with coupling width and varied the excitatory/inhibitory balance from to , thereby moving the system from the oscillatory through the chaotic into the fixed-point regime. As expected, the covariance measure shifts from a plateau near , through one around 0, to one near (Figure 8f). The covariance computed only from diagonal terms (orange) follows the same trend as that based on off-diagonal (green) or all terms combined (blue).
4 Discussion
In this work, we investigated how three distinct organizational regularities—Hopfield reciprocity, Dale homogeneity, and modularity—affect the dynamical behavior and computational performance of recurrent neural networks (RNNs).
4.1 Prior expectations and numerical findings
4.1.1 Hopfield reciprocity
Hopfield-type reciprocity introduces symmetry into the weight matrix by increasing the probability that any connection from neuron A to B is mirrored by an equal connection from B to A. In classical Hopfield networks, such symmetry is instrumental in stabilizing fixed point attractors, corresponding to stored patterns. With some notable exceptions (), these networks normally use binary units and asynchronous updates, enabling the system to settle gradually into one of the memorized configurations. The symmetric weights ensure that the energy landscape has defined minima, guiding the dynamics toward stable fixed points.
In our model, the situation differs in several respects. The neurons are continuous-valued with activation, and all updates occur synchronously. Nonetheless, it was natural to expect that increasing Hopfield reciprocity might exert an ordering influence on the network dynamics - possibly reducing chaotic fluctuations and thus enhancing information processing capabilities.
The numerical results only partially confirm this expectation. As increases, the most consistent effect across the three phase points is a strong rise in nonlinearity . This indicates that the reservoir is increasingly driven into a saturated, quasi-digital regime, which is generally disadvantageous for many computational tasks. The neural activations reveal subpopulations of neurons that settle into fixed points, while others exhibit period-two oscillations. Moreover, the instantaneous Pearson correlation matrices (Figures 7d–f) show that reciprocal coupling induces strongly correlated neuron pairs or groups, often displaying nearly perfect positive or negative correlations that persist even under added noise. Overall, these effects tend to reduce the computational performance.
4.1.2 Dale homogeneity
Dale’s principle, a key feature of biological neural networks, stipulates that each neuron maintains a fixed output polarity—either excitatory or inhibitory—across all its targets. In our model, this principle is implemented by the homogeneity parameter , which controls the consistency of output signs within each column of the weight matrix. At , neurons send mixed excitatory and inhibitory outputs; at , every neuron acts strictly as a monopolar sender.
From a theoretical perspective, one might expect that Dale homogeneity introduces a more directional and interpretable signal flow through the reservoir. Specifically, we anticipated that increasing would suppress high-frequency or erratic fluctuations by enforcing more coherent influence patterns among neurons, which might also be reflected in their mutual instantaneous correlations.
Indeed, as the homogeneity parameter increases, the initially very small instantaneous Pearson correlations observed in the chaotic phase point C are replaced by much stronger, predominantly positive correlations. An unexpected feature was that neural activations now exhibit quasi-periodic collective oscillations with longer periods and noticeable phase shifts between individual neurons. Nevertheless, the accuracy in our test task clearly improves when Dale regularity is introduced.
4.1.3 Modularity (Block size )
In the case , the modularity parameter does not produce distinct structural blocks in the weight matrix. Instead, increasing gradually transforms the underlying weight distribution from a single Gaussian to a mixture of two Gaussians—one narrower (weaker weights) and one broader (stronger weights)—with the total standard deviation kept constant. This results in a heterogeneous distribution of connection strengths with a much broader tail.
We expected that such broader-tailed weight distributions might have a beneficial effect on the network’s information-processing capacity. In biological neural circuits, especially within the human cortex, synaptic connection strengths are known to follow heavy-tailed, approximately log-normal distributions, where a small subset of strong connections coexists with a majority of weak ones (). This structural heterogeneity has been suggested to support both robustness and dynamic richness by combining stable, high-impact pathways with a flexible background of weaker connections.
Indeed, we find in our simulations a clear rise in task accuracy , despite the absence of actual modular structure.
4.1.4 Modularity (Block size )
In biological neural networks, modular organization is a well-established principle observed across spatial scales—from cortical microcircuits to large brain regions. Modules, typically defined as groups of neurons with strong internal connectivity and weaker coupling to the rest of the network, are thought to support functional specialization while preserving global integration. From this perspective, introducing modularity into artificial RNNs could plausibly enhance their computational capacity by enabling localized processing and buffering against global instabilities.
In our model, modularity is implemented via the control parameter , which adjusts the variance of intra- and inter-module connection strengths while keeping the overall standard deviation constant. When the block size is set to , the resulting matrix exhibits clearly defined patches of stronger weights embedded in a weaker background. However, these high-variance patches may lie on or off the diagonal. Diagonal modules imply recurrently coupled local groups, whereas off-diagonal modules encode directional connections from one group of neurons to another, without necessarily reciprocal feedback. While the former can stabilize local loops of activity, the latter may implement feedforward-like interactions across functionally distinct subnetworks. Though less intuitively interpretable, these asymmetric modules still contribute to shaping constrained, layered dynamics within the recurrent architecture.
Prior to numerical investigation, we expected that such modular organization could lead to partial functional segregation, a dampening of chaotic fluctuations and possibly more reproducible activity patterns within modules.
Indeed, these expectations are supported by the simulations. As increases, we observe a pronounced reduction in both fluctuation and nonlinearity , indicating that the reservoir dynamics become more stable and less saturated. This effect is especially clear in balanced networks , which otherwise tend to exhibit chaotic behavior. The drop in suggests that spontaneous overactivation is strongly suppressed, while the lower points to a shift away from digital-like saturation toward a more linear or intermediate regime.
The Pearson correlation matrices (Figures 7j–l) provide a complementary view of this effect: they reveal substructures of internally coherent neuron clusters with weaker, mixed-sign couplings between modules. The correlations are neither too weak (as in chaotic regimes) nor too strong (as in fixed point or short-period states) but distributed across the full range. This balanced structure appears to create particularly favorable conditions for reservoir computing.
Most strikingly, both in reservoirs of 50 and 100 neurons, the accuracy in the sequence generation task increases sharply and reaches near-maximal levels across a wide range of . The improvement in computational performance surpasses that achieved with Dale homogeneity. A very similar general effect of this organizational regularity was found in the patches classification task.
4.2 Work limitations and future perspectives
The present study has focused almost exclusively on one specific computational task: the generation of predefined output sequences from class-specific input stimuli. While this task is well-suited for evaluating the internal stability and reproducibility of reservoir trajectories, it represents only one of many possible functional challenges an RNN might face. Future studies will systematically explore how the three structural regularities—Hopfield reciprocity, Dale homogeneity, and modularity—affect other task types, including classification, prediction, temporal integration, and generative modeling. These tasks may impose different demands on the reservoir, potentially favoring entirely different dynamical regimes.
Several further limitations of the current setup should be noted. Our networks use a fixed, pointwise activation function throughout, with no mechanism for adapting nonlinear response properties during training or evolution. Moreover, the recurrent connections are entirely fixed, without plasticity or learning mechanisms. While this simplification allows us to isolate the impact of architectural regularities, it remains unclear how the observed effects extend to systems with ongoing synaptic adaptation. Likewise, our networks are relatively small and shallow, lacking hierarchical depth or multi-scale processing pathways. It is an open question whether similar regularities exert comparable influences in large-scale architectures with layered structure, where modularity and reciprocity might interact differently with gradient propagation and representational abstraction.
A further important direction concerns the interaction between structural regularities. In this study, we varied each regularity parameter in isolation while keeping the others fixed. However, real biological systems typically exhibit several regularities at once. It remains an open question whether combinations of regularities act synergistically or interfere with each other. For instance, it is conceivable that modularity and Dale homogeneity together enhance performance more than either alone—or that reciprocity counteracts the benefits of modular organization.
A separate line of inquiry could explore the relevance of the observed effects for biological computation, although in this context, it would be more realistic to interpret each of our RNN units not a single neuron, but as a homogeneous neuronal assembly of a cortical network (; ). For example, it could be further investigated whether biological circuits show anything resembling Hopfield symmetry at the level of neuronal assemblies. So far, while exact synaptic reciprocity is not observed, cortical microcircuits consistently exhibit an overrepresentation of bidirectional connections (; ) and population dynamics with attractor-like stability. These features suggest that, on a coarse-grained assembly level, biological networks may display approximate or statistical reciprocity, even though the precise Hopfield topology is not realized.
Since the structural regularities studied here are motivated by neurobiological observations, it is worthwhile to compare their dynamical implications to actual brain circuits. This could involve applying the same dynamical and performance metrics to empirically derived connectomes—such as those of C. elegans, Drosophila, or the zebrafish larva—and seeing how they perform on comparable tasks.
Beyond empirical validation, the theoretical understanding of how structural regularities shape network dynamics remains incomplete. For instance, it is still unclear why modularity so reliably suppresses fluctuation and nonlinearity, or why Dale homogeneity improves accuracy without driving the system into linearity. While it may be tempting to invoke classical tools such as the spectral radius or eigenvalue spectra of the weight matrix, such linear measures often fail to capture the complex behavior of nonlinear, recurrent systems. In our view, a more fruitful approach would be to characterize how the structural parameters influence the geometry of the state space, the stability of trajectories, or the repeatability of state sequences under repeated input. Concepts such as state convergence, divergence under perturbations, and the reproducibility of internal trajectories may provide more robust and interpretable metrics than traditional eigenvalue-based criteria. A related approach was proposed by Legenstein and Maass (), who linked the computational performance of neural microcircuits to their dynamical regime near the edge of chaos, using measures of kernel quality and generalization capability to predict functional performance. Developing such diagnostics could help formulate a more general understanding of how structural constraints give rise to functional dynamics in RNNs.
Another promising direction lies in allowing structural regularities to vary dynamically over time. Instead of statically imposed homogeneity or modularity, one could investigate networks in which these properties emerge or change through learning or adaptation. This would connect structural regularities more directly to plasticity rules and functional demands, potentially offering new models of task-dependent reconfiguration in neural circuits.
Statements
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Author contributions
CM: Writing – original draft, Visualization, Investigation, Methodology, Formal Analysis, Validation, Conceptualization, Data curation. AS: Project administration, Funding acquisition, Supervision, Writing – original draft, Conceptualization. AM: Resources, Writing – original draft, Data curation, Validation, Funding acquisition. PK: Methodology, Writing – original draft, Supervision, Investigation, Conceptualization, Funding acquisition, Project administration, Validation, Resources.
Funding
The author(s) declared that financial support was received for this work and/or its publication. This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation): grants KR 5148/3-1 (project number 510395418), KR 5148/5-1 (project number 542747151), KR 5148/10-1 (project number 563909707) and GRK 2839 (project number 468527017) to PK, and grants SCHI 1482/3-1 (project number 451810794) and SCHI 1482/6-1 (project number 563909707) to AS.
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.
Generative AI statement
The author(s) declared that generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AguiarM.DasA.JohanssonK. H. (2023). “Universal approximation of flows of control systems by recurrent neural networks,” in 2023 62nd IEEE conference on decision and control (CDC) (IEEE), 2320–2327.
2
AlzubaidiL.ZhangJ.HumaidiA. J.Al-DujailiA.DuanY.Al-ShammaO.et al (2021). Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. J. Big Data8 (1), 1–74. 10.1186/s40537-021-00444-8
3
BarakO. (2017). Recurrent neural networks as versatile tools of neuroscience research. Curr. Opinion Neurobiology46, 1–6. 10.1016/j.conb.2017.06.003
4
BerlemontK.MongilloG. (2022). Glassy phase in dynamically-balanced neuronal networks, 2022–2023. bioRxiv.
5
BönselF.KraussP.MetznerC.YamakouM. E. (2021). Control of noise-induced coherent oscillations in time-delayed neural motifs. arXiv preprint arXiv:2106.11361.
6
BrunelN. (2016). Is cortical connectivity optimized for storing information?Nat. Neuroscience19 (5), 749–755. 10.1038/nn.4286
7
BüsingL.SchrauwenB.LegensteinR. (2010). Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Computation22 (5), 1272–1311. 10.1162/neco.2009.01-09-947
8
CornfordJ.KalajdzievskiD.LeiteM.LamarquetteA.KullmannD. M.RichardsB. (2020). Learning to live with Dale’s principle: ANNs with separate excitatory and inhibitory units. BioRxiv, 2011–2020. 10.1101/2020.11.02.364968
9
CucchiM.AbreuS.CicconeG.BrunnerD.KleemannH. (2022). Hands-on reservoir computing: a tutorial for practical implementation. Neuromorphic Comput. Eng.2 (3), 032002. 10.1088/2634-4386/ac7db7
10
DambreJ.VerstraetenD.SchrauwenB.MassarS. (2012). Information processing capacity of dynamical systems. Sci. Reports2 (1), 1–7. 10.1038/srep00514
11
FarrellM.RecanatesiS.MooreT.LajoieG.Shea-BrownE. (2022). Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. Nat. Mach. Intell.4 (6), 564–573. 10.1038/s42256-022-00498-0
12
FelixZ.TrieschJ. (2017). Hoffmann and jochen triesch. Nonrandom network connectivity comes in pairs. Netw. Neuroscience1 (1), 31–41. 10.1162/NETN_a_00004
13
FolliV.GostiG.LeonettiM.RuoccoG. (2018). Effect of dilution in asymmetric recurrent neural networks. Neural Netw.104, 50–59. 10.1016/j.neunet.2018.04.003
14
FournierS. J.PaccoA.RosV.UrbaniP. (2025). Non-reciprocal interactions and high-dimensional chaos: comparing dynamics and statistics of equilibria in a solvable model. arXiv preprint arXiv:2503.20908.
15
GerumR. C.ErpenbeckA.KraussP.SchillingA. (2020). Sparsity through evolutionary pruning prevents neuronal networks from overfitting. Neural Netw.128, 305–312. 10.1016/j.neunet.2020.05.007
16
GononL.OrtegaJ.-P. (2021). Fading memory echo state networks are universal. Neural Netw.138, 10–13. 10.1016/j.neunet.2021.01.025
17
HavivD.RivkindA.BarakO. (2019). “Understanding and controlling memory in recurrent neural networks,” in International conference on machine learning. Cambridge, MA: Proceedings of Machine Learning Research (PMLR), 2663–2671.
18
HintonG. E.SabourS.FrosstN. (2018). “Matrix capsules with em routing,” in International Conference on Learning Representations (ICLR), Vancouver, BC, Canada, April 30 - May 3, 2018.
19
HwangS.FolliV.LanzaE.ParisiG.RuoccoG.ZamponiF. (2019). On the number of limit cycles in asymmetric neural networks. J. Stat. Mech. Theory Exp.2019 (5), 053402. 10.1088/1742-5468/ab11e3
20
HwangS.LanzaE.ParisiG.RocchiJ.RuoccoG.ZamponiF. (2020). On the number of limit cycles in diluted neural networks. J. Stat. Phys.181 (6), 2304–2321. 10.1007/s10955-020-02664-3
21
IkemotoS.DallaLiberaF.KohH. (2018). Noise-modulated neural networks as an application of stochastic resonance. Neurocomputing277, 29–37. 10.1016/j.neucom.2016.12.111
22
JaegerH. (2001). The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology. GMD Technical Report13.
23
JaegerH. (2014). Controlling recurrent neural networks by conceptors. arXiv preprint arXiv:1403.3369.
24
KnightB. W. (2000). Dynamics of encoding in neuron populations: some general mathematical features. Neural Comput.12 (3), 473–518. 10.1162/089976600300015673
25
KraussP.TziridisK.MetznerC.SchillingA.UlrichH.SchulzeH. (2016). Stochastic resonance controlled upregulation of internal noise after hearing loss as a putative cause of tinnitus-related neuronal hyperactivity. Front. Neuroscience10, 597. 10.3389/fnins.2016.00597
26
KraussP.PrebeckK.SchillingA.MetznerC. (2019a). “Recurrence resonance” in three-neuron motifs. Front. Computational Neuroscience13, 64. 10.3389/fncom.2019.00064
27
KraussP.ZanklA.SchillingA.SchulzeH.MetznerC. (2019b). Analysis of structure and dynamics in three-neuron motifs. Front. Comput. Neurosci.13 (5), 5. 10.3389/fncom.2019.00005
28
KraussP.SchusterM.DietrichV.SchillingA.SchulzeH.MetznerC. (2019c). Weight statistics controls dynamics in recurrent neural networks. PloS One14 (4), e0214541. 10.1371/journal.pone.0214541
29
KühnR.BösS. (1993). Statistical mechanics for neural networks with continuous-time dynamics. J. Phys. A Math. General26 (4), 831–857. 10.1088/0305-4470/26/4/012
30
LeCunY.BengioY.HintonG. (2015). Deep learning. Nature521 (7553), 436–444. 10.1038/nature14539
31
LegensteinR.MaassW. (2007). Edge of chaos and prediction of computational performance for neural circuit models. Neural Networks20 (3), 323–334. 10.1016/j.neunet.2007.04.017
32
LutzM.SchuchhardtJ.SchusterH. G. (1992). Suppressing chaos in neural networks by noise. Phys. Review Letters69 (26), 3717–3719. 10.1103/PhysRevLett.69.3717
33
MaheswaranathanN.WilliamsA. H.GolubM. D.GanguliS.SussilloD. (2019). Universality and individuality in neural dynamics across large populations of recurrent networks. Adv. Neural Information Processing Systems2019, 15629–15641. 10.48550/arXiv.1907.08549
34
MattiaM.Del GiudiceP. (2002). Population dynamics of interacting spiking neurons. Phys. Rev. E66 (5), 051917. 10.1103/PhysRevE.66.051917
35
Maximilian SchäferA.ZimmermannH. G. (2006). “Recurrent neural networks are universal approximators,” in International conference on artificial neural networks (Springer), 632–640.
36
MetznerC.KraussP. (2022). Dynamics and information import in recurrent neural networks. Front. Comput. Neurosci.16, 876315. 10.3389/fncom.2022.876315
37
MetznerC.SchillingA.MaierA.KraussP. (2024). Recurrence resonance-noise-enhanced dynamics in recurrent neural networks. Front. Complex Syst.2, 1479417. 10.3389/fcpxs.2024.1479417
38
MetznerC.SchillingA.MaierA.KraussP. (2025). Nonlinear neural dynamics and classification accuracy in reservoir computing. Neural Comput.37 (8), 1469–1504. 10.1162/neco_a_01770
39
MeunierD.LambiotteR.BullmoreE. T. (2010). Modular and hierarchically modular organization of brain networks. Front. Neurosci.4, 200. 10.3389/fnins.2010.00200
40
MinB.RossH.SulemE.VeysehA. P. B.NguyenT. H.SainzO.et al (2023). Recent advances in natural language processing via large pre-trained language models: a survey. ACM Comput. Surv.56 (2), 1–40. 10.1145/3605943
41
NarangS.ElsenE.DiamosG.SenguptaS. (2017). Exploring sparsity in recurrent neural networks. arXiv preprint arXiv:1704.05119.
42
PerinR.BergerT. K.MarkramH. (2011). A synaptic organizing principle for cortical neuronal groups. Proc. Natl. Acad. Sci.108 (13), 5419–5424. 10.1073/pnas.1016051108
43
RajanK.AbbottL. F.SompolinskyH. (2010). Stimulus-dependent suppression of chaos in recurrent neural networks. Phys. Rev. E82 (1), 011903. 10.1103/PhysRevE.82.011903
44
RodriguezN.IzquierdoE.AhnY-. Y. (2019). Optimal modularity and memory capacity of neural reservoirs. Netw. Neurosci.3 (2), 551–566. 10.1162/netn_a_00080
45
RosenbaumC.KlingerT.RishI. (2018). “Routing networks: adaptive selection of non-linear functions for multi-task learning,” in International Conference on Learning Representations (ICLR), Vancouver, BC, Canada, April 30 - May 3, 2018.
46
RosenbaumC.RishI.KlingerT. (2019). Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv:1904.12574.
47
SabourS.FrosstN.HintonG. E. (2017). Dynamic routing between capsules. Adv. Neural Inf. Process. Syst.30. 10.48550/arXiv.1710.09829
48
SchillingA.TziridisK.SchulzeH.KraussP. (2021). The stochastic resonance model of auditory perception: a unified explanation of tinnitus development, zwicker tone illusion, and residual inhibition. Prog. Brain Res.262, 139–157. 10.1016/bs.pbr.2021.01.025
49
SchillingA.GerumR.MetznerC.MaierA.KraussP. (2022). Intrinsic noise improves speech recognition in a computational model of the auditory pathway. Front. Neurosci.16, 908330. 10.3389/fnins.2022.908330
50
SchillingA.SedleyW.GerumR.MetznerC.TziridisK.MaierA.et al (2023). Predictive coding and stochastic resonance as fundamental principles of auditory phantom perception. Brain146 (12), 4809–4825. 10.1093/brain/awad255
51
SchueckerJ.GoedekeS.HeliasM. (2018). Optimal sequence memory in driven random networks. Phys. Rev. X8 (4), 041029. 10.1103/physrevx.8.041029
52
ShazeerN.MirhoseiniA.MaziarzK.DavisA.LeQ.HintonG.et al (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint
53
SomogyiP.TamásG.LujánR.BuhlE. H. (1998). Salient features of synaptic organisation in the cerebral cortex. Brain Res. Rev.26 (2-3), 113–135. 10.1016/S0165-0173(97)00061-1
54
SongS.Jesper SjöströmP.ReiglM.NelsonS.ChklovskiiD. B. (2005). Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS Biology3 (3), e68. 10.1371/journal.pbio.0030068
55
SpornsO.BetzelR. F. (2016). Modular brain networks. Annu. Rev. Psychol.67, 613–640. 10.1146/annurev-psych-122414-033634
56
StrataP.HarveyR. (1999). Dale’s principle. Brain Res. Bull.50 (5-6), 349–350. 10.1016/s0361-9230(99)00100-8
57
WallaceE.MaeiH. R.LathamP. E. (2013). Randomly connected networks have short temporal memory. Neural Computation25 (6), 1408–1439. 10.1162/NECO_a_00449
58
ZadorA. M. (2019). A critique of pure learning and what artificial neural networks can learn from animal brains. Nat. Commun.10 (1), 3770. 10.1038/s41467-019-11786-6
Summary
Keywords
dale’s principle, hopfield network, modularity, recurrent neural networks, reservoir computing
Citation
Metzner C, Schilling A, Maier A and Krauss P (2026) Organizational regularities in recurrent neural networks. Front. Complex Syst. 3:1636222. doi: 10.3389/fcpxs.2025.1636222
Received
27 May 2025
Revised
28 November 2025
Accepted
30 November 2025
Published
05 January 2026
Volume
3 - 2025
Edited by
Claudio Castellano, Istituto dei Sistemi Complessi (ISC-CNR), Italy
Reviewed by
Gabriele Di Antonio, Santa Lucia Foundation, Italy
Tobias Kühn, University of Bern, Switzerland
Updates
Copyright
© 2026 Metzner, Schilling, Maier and Krauss.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Patrick Krauss, patrick.krauss@fau.de
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.